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Preface 


Over  the  past  ten  years  Software  Reliability  has 
emerged  as  a new  discipline.  It  has  gained  attention 
because  of  the  vast  amount  of  money  being  spent  annually  on 
computer  software  by  private  and  governmental  agencies. 
Numerous  models  to  predict  software  reliability  and  its 
attributes  have  been  presented  by  various  individuals  and 
organizations . One  of  the  biggest  setbacks  in  the 
verification  of  these  models  has  been  the  lack  of  software 
error  data  bases.  Recognizing  the  importance  of  this 
subject,  Rome  Air  Development  Center  (RADC/ISIS)  at  Griffiss 
AFB,  N.Y.  contracted  several  companies  to  collect  software 
error  data  from  software  projects  developed  for  the 
government.  RADC  has  now  built  a large  repository  of  data. 
Utilizing  files  from  the  RADC  repository,  discrimination 
between  several  software  reliability  models  by  hypothesis 
testing  is  performed.  A computer  program  is  presented  which 
will  predict  the  best  software  reliability  model  for 
particular  data.  Time-to-error  data  was  used  in  the 
software  reliability  model  testing  and  time-to-fix  data  was 
used  in  the  modelling  of  time-to-fix.  Due  to  limitations  of 
the  data,  some  of  the  software  reliability  models  considered 
could  not  be  tested.  For  the  models  that  could  be  tested, 
computer  program  subroutines  were  written  to  estimate  the 
parameters  of  the  software  reliability  models.  After 
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parameter  estimation  was  completed,  the  Likelihood  Ratio 
Test  was  used  for  hypothesis  testing  to  determine  which 
model  best  suited  the  particular  data  base  being  tested. 
Also,  model  testing  was  implemented  for  the  time-to-fix  data 
using  eleven  of  the  more  commonly  used  probability  density 
functions,  the  Kolmogorov-Smirnov  Test,  and  the  Likelihood 
Ratio  Test. 

The  theoretical  foundation  of  this  thesis  is 
comprised  of  reliability  theory,  statistics,  and  computer 
software  development.  However,  this  thesis  avoids  going 
into  too  much  detail  concerning  statistical  and  software 
reliability  theory  and  formulation.  Rather,  a complete 
bibliography  is  provided  for  those  desiring  further  details 
on  development  of  formulas  and  mathematical  techniques. 
With  one  exception  the  actual  code  of  the  computer  programs 
and  much  of  the  output  produced  for  thesis  will  not  be 
presented.  The  contribution  of  these  listings  would  be 
negligible  when  compared  to  their  volume.  The  computer 
source  code  used  in  parameter  estimation  and  discrimination 
between  software  reliability  models  is  included.  Usage  of 
this  program  requires  only  knowledge  of  the  difference  in 
time  between  errors  and  the  number  of  errors  occurring 
during  each  interval. 

A project  such  as  this  cannot  be  done  without  moral 
and  technical  support  from  others.  I thank  Dr.  Thadeus  L. 
Regulinski,  my  thesis  advisor,  for  his  invaluable  guidance, 
assistance,  and  his  willingness  to  share  his  academic  and 
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technical  knowledge.  Also,  I appreciate  the  assistance  that 
I received  from  Dr.  Paul  Moranda  of  the  McDonnell  Douglas; 
Dr.  E.  Nelson,  Mr.  Tom  Thayer,  and  Mr.  Myron  Lipow  of  TRW; 
Mr.  H.  E.  Willman  of  Raytheon  Company;  my  thesis  sponsor, 
Mr.  Allen  Sukert  of  RADC;  and  my  thesis  readers,  Capt. 
Edward  Reeves  and  Capt.  James  Peterson.  A day  did  not  go  by 
where  my  thesis  did  not  effect  my  family,  whether  it  be  a 
late  night  at  the  computer  center  or  asking  for  quiet  in  the 
house.  I would  like  to  give  special  thanks  to  my  wife  and 
children  for  their  encouragement,  understanding,  and  most  of 
all  patience. 
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Abstract 


Various  competing  software  reliability  models  were 

Wit 

tested  ttt-ii4-2-in^^time-to-error  data  extracted  from  a large 
data  base  provided  by  Rome  Air  Development  Centei*^ 

C N-  /- 

Griffis  Mr  Force  Base,  New  York.>^  Only  those  models  for 
which  appropriate  information  was  available  from  the  RADC 
data  base  were  used.  Each  model  that  was  tested  had  its 
parameters  estimated  and  discrimination  between  the  models 
was  performed  by  the  Likelihood  Ratio  Test.  Code  for  the 
computer  program  used  in  parameter  estimation  and  Likelihood 
Ratio  Testing  is  included.  The  same  large  data  base  was 
also  used  in  determination  of  the  probability  density 
functions  governing  the  random  variable  time-to-fix.  Time- 
to-fix  data  was  extracted  from  the  data  base  and  was  sorted 


by  error  categories.  Eleven  of  the  more  common  probability 
density  functions,  along  with  the  Kolmogorov-Smirnov  Test 
and  the  Likelihood  Ratio  Test,  were  used  to  isolate  the 
d i s t:' ibut  ion  that  best  modelled  software  time-to-fix  for 
each  particular  data  se:. 


SOFTWARE  RELIABILITY:  MODELLING 
TIME-TO-ERROR  AND  TIME-TO-FIX 


Chapter  _I 
Introduction 


Each  year  billions  of  dollars  are  being  spent  on  the 
contractual  purchase  and  in-house  development  of  computer 
software  packages.  The  future  does  not  look  " ich  brighter, 
especially  considering  that  the  cost  of  software  is  soaring. 
The  cost  of  planning,  coding,  verifying  and  maintaining 
software  has  far  exceeded  the  cost  of  computer  hardware. 
Unfortunately,  more  often  than  not,  the  final  software 
product  is  far  too  unreliable. 


"The  world's  most  carefully  planned 
and  generously  funded  software  program 
was  that  developed  for  the  Apollo  series 
of  lunar  flights.  The  effort  attracted 
some  of  the  Nation's  best  computer 
programmers  and  involved  two  competing 
teams  checking  the  software  as 
thoroughly  as  the  experts  knew  how  to 
make  it.  In  the  aggregate  about  $600 
million  was  spent  on  software  for  the 
Apollo  program.  Yet,  almost  every  major 
fault  of  the  Apollo  program,  from  false 
alarms  to  actual  mishaps,  was  the  direct 
result  of  errors  in  computer  software 
(Ref  [26]:52)." 


This  thesis  addresses  the  problem  of  unreliable 
software.  The  first  objective  of  the  thesis  is  to  test  the 
numerous  competing  software  reliability  models.  Many  models 
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have  been  presented  by  various  authors.  Each  has  claimed 
his  model  to  be  the  one  that  descibes  software  reliability 
best.  Most  of  these  models  were  formulated  on  the  basis  of 
small  data  bases.  A large  software  error  data  base  provided 
for  use  in  this  thesis  by  Rome  Air  Development  Center 
(RADC),  Griffiss  AFB,  N.  Y.  permitted  testing  of  the  models. 

The  second  objective  is  to  determine  which,  if  any, 
probability  density  functions  (pdf)  govern  time-to-fix  for 
entire  error  files  and  error  categories  within  files.  As  far 
as  can  be  determined  no  such  analysis  has  been  previously 
performed.  Further  discussion  of  these  objectives  will  be 
enumerated  in  later  chapters,  but  first  the  basics  of 
reliable  software,  software  reliability,  and  software  time- 
to-fix  must  be  understood. 

What  is  reliable  software  and  how  can  it  be 
produced?  This  is  a very  important  question.  The  first 
part  of  the  question  can  be  answered  fairly  simply  and  the 
answer  can  be  agreed  upon  by  the  majority  of  software 
engineers.  "Reliable  software"  is  considered  to  be  software 
that  produces  output  within  specified  tolerance  limits  over 
a specified  period  of  time.  On  the  other  hand,  a "software 
error"  is  considered  to  be  any  software  condition  which 
causes  the  output  to  deviate  from  the  specified  tolerances. 
"Software  reliability"  is  the  probability  that  the  software 
will  perform  within  the  specified  requirements  over  a 
specified  time  interval  ( Ref [24 ] : 5-2 ) . The  answer  to  the 
second  part  of  the  question  is  not  so  easily  agreed  upon. 
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There  are  several  schools  of  thought  being  promoted  on  how 
to  produce  reliable  software.  One  school  believes  strongly 
in  better  development  techniques  as  the  key  to  more  reliable 
software.  Structured  programming,  top  down  design  and 
modularity  are  terms  that  are  expounded  by  followers  of  this 
school.  Another  school  proposes  better  testing  and  proof- 
of-correctness  techniques  as  the  answer.  Computer  hardware 
has  long  had  the  capability  of  fault  tolerance,  the  state  of 
being  able  to  recover  from  an  error  condition.  Fault 
tolerance  is  now  being  considered  in  the  software 
engineering  field  as  a viable  means  of  obtaining  reliable 
software  (Ref[ 17] :230-232) . Undoubtedly,  each  school  of 
thought  has  something  significant  to  offer  and  a manager  of 
a software  project  must  consider  all  options  when  purchasing 
or  producing  software. 

Despite  the  best  laid  plans  and  procedures  as 
promoted  by  the  various  schools  of  thought,  a software 
package  with  no  errors  is  a rarity.  A program  can  be 
repeatedly  tested,  but  "Testing  only  shows  the  presence  of 
errors,  not  the  absence  (Ref[6] : 33)” • In  the  past  the 
software  manager  has  had  no  data  to  determine  whether  or  not 
a new  software  system  is  reliable  enough  to  be  implemented, 
other  than  the  instinct  of  a programmer  or  the  assurance  of 

t 

a software  contractor.  How  does  he  know  that  he  has  not 
purchased  a nightmare?  He  has  had  no  means  of  estimating 
the  reliability  of  the  software  package.  It  was  a simple 
matter  to  define  software  reliability,  but  it  is  quite 


another  thing  to  measure  it.  Even  if  there  was  a way  to 
measure  the  reliability  and  the  manager  decided  to  accept 
the  software  system,  how  does  he  determine  the  mean-time-to- 
fix  after  the  software  system  has  been  delivered?  In  order 
to  ascertain  the  answers  to  these  questions  the  software 
manager  needs  to  know  the  basic  principles  of  software 
reliability.  Software  reliability  could  someday  easily  play 
an  important  role  in  his  decision  making  policy.  But,  it 
was  not  until  recent  years  that  attempts  were  made  to 
mathematically  descibe  software  reliability.  Some  of  the 
basic  principles  of  software  reliability  are  still  being 
disputed  by  software  engineers.  A closer  look  into  its 


history  and  background  may  help  explain  this  fact. 


Software  reliability  has  emerged  as  a discipline  of 
its  own  only  in  the  past  decade.  Early  attempts  to  model 
software  were  based  on  hardware  principles.  The  "bathtub" 
curve  (Fig.  1)  is  a model  often  used  in  demonstrating 
hardware  reliability.  The  curve  can  be  divided  into  three 
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phases;  burn-in,  normal-life,  and  wearout.  During  burn-in 
the  failure  rate  is  decreasing  with  time;  during  the  normal- 
life  phase  the  failure  rate  is  constant  with  time;  and 
during  wear-out  the  failure  rate  is  increasing  with  time. 
As  time  increases  over  the  life  of  a software  package,  the 
number  of  errors  found  per  unit  of  time  (detection  rate) 
should  decrease.  This  most  closely  resembles  the  burn-in 
phase.  But,  It  has  been  shown  in  at  least  two  projects  that 
the  number  of  new  errors  introduced  during  correction 
procedures  is  not  trivial  (Ref  [7]:18,  [24]:^-'45).  Some 

software  reliability  models  assume  there  are  no  new  errors 
introduced.  Before  any  criticism  is  waged  against  these 
models,  the  environment  under  which  they  were  developed 
should  be  recognized. 

Collection  of  software  errors  and  their  attributes 
has  only  taken  place  in  recent  years.  Many  of  the  models 
had  to  be  developed  and  validated  using  small  data  bases. 
There  were  no  large  software  error  data  bases  to  .use  for 
analysis.  This  brings  out  an  interesting  question.  From  the 
large  number  of  models  available,  how  does  the  software 
manager  know  which  one  to  choose  for  his  project?  Is  one 
model  better  than  all  others  for  all  data  or  are  particular 
models  suited  for  particular  situations?  These  are* 
questions  that  have  to  be  answered  before  any  faith  can  be 
placed  in  software  reliability  models.  These  questions  will 
be  answered  in  the  process  of  meeting  the  objectives  of  this 
thesis.  However,  a large  data  base  was  required  before  any 
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reasonable  answers  could  be  obtained.  In  an  attempt  to  get 

V 

full  value  from  its  software  dollar,  the  Air  Force  has 

^ established  a software  error  repository  at  Rome  Air 

Development  Center  (RADC),  Griffiss  AFB,  New  York.  The 

repository  presently  consists  of  over  27,000  software  error 
reports  accumulated  from  at  least  seven  different  Air  Force 
projects.  Each  error  record  within  the  repository  contains 
attributes  such  as  date-of-error , date-of-fix,  error 

category,  and  error  criticality.  This  thesis  is  based  upon 
[ the  above  data. 

I 

Chapter  2 describes  the  characteristics  and 
background  of  the  different  files  from  the  repository.  These 
files  are  the  best  that  are  available  at  the  present  time  to 
test  the  software  reliability  models  and  to  determine  the 
pdfs  governing  time-to-fix  for  software  error  categories. 
Chapter  3 identifies  software  reliability  models  that  were 
considered  for  testing  purposes  and  delineates  the  model 
selection  criteria.  Details  on  the  time-to-fix  validation 
program,  which  was  used  in  the  selection  of  the  governing 
pdfs  of  time-to-fix,  are  also  presented.  Chapter  4 includes 
information  on  how  the  data  from  the  data  repository  was 
extracted  and  prepared  for  usage  by  the  software  reliability 
models  and  the  time-to-fix  program.  Limitations  of  the  data 
files  are  also  pointed  out.  Chapter  5 outlines  the  program 
construction  process  that  was  necessary  to  validate  and 
discriminate  between  the  usable  software  reliability  models. 
Chapter  6 summarizes  the  enormous  amount  of  computer  output 
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that  was  generated  by  this  study  and  presents  the  results  In 
condensed  form.  A summary  is  presented  of  the  results  from 
the  parameter  estimation  and  hypothesis  testing  of  the 
software  reliability  models.  Summaries  of  the  distributions 
selected  for  time-to-fix  are  also  presented.  In  support  of 
the  objectives  set  forth  earlier,  the  conclusions  and 
recommendations  that  were  derived  from  the  results  are 
contained  in  Chapter  7. 
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Data  Description 


II. 1 Basic  Characteristics 

Man's  reluctance  to  admit  error  may  be  one  reason 
why  software  error  data  has  not  been  collected  earlier. 
Besides,  the  computer  has  been  the  best  scapegoat  mankind 
has  had  in  a long  time.  It's  been  too  easy  to  say,  "The 
computer  did  it."  If  the  ever-rising  cost  of  software  is  to 
be  halted,  man  must  admit  that  the  mistakes  were  his  and  not 
those  of  some  mythical , error-generating  computer. 

There  was  no  precedence  for  collecting  a large 
volume  of  software  errors  before,  and  the  RADC  files  mark  a 
significant  beginning.  Most  assuredly,  improvements  can  be 
made  in  data  collection  procedures,  and  each  error- 
collecting contractor  made  many  such  recommendations.  In 
fact  a new,  shorter,  and  more  meaningful  error  category  list 
has  since  been  established  (Ref  [24]:  3-18  thru  3-20). 

The  selection  of  software  reliability  models  to  be 
tested  was  largely  dependent  on  the  data  available.  Not  all 
software  reliability  models  could  be  tested  because  the  data 
did  not  lend  itself  to  them.  Consequently,  before  any 
discussion  concerning  models  is  presented,  a basic 
understanding  of  the  data  is  required. 

As  mentioned  earlier,  six  data  files  from  the  RADC 


8 


repository  were  used  as  input  data.  The  six  files  represent 
27,000  software  errors  from  five  different  projects  by  four 
different  contractors.  Each  record  within  each  file 
represents  a Software  Problem  Report(SPR).  With  a few 
exceptions,  each  record  within  a file  consists  of  the 
following  information: 

1)  Date  software  error  report  was  opened 

2)  Date  software  error  report  was  closed 

3)  Error  category 

4)  Degree  of  criticality 

5)  Test  phase  when  error  occurred 

6)  Module  in  which  error  occurred. 

One  of  the  most  important  aspects  to  realize  about 
the  data  is  that  it  was  not  collected  solely  for  the  purpose 
for  which  it  is  being  used  in  this  investigation.  The  data 
is  being  accepted  as  the  best  data  available.  The  error 
categories  used  in  the  collection  of  this  data  are  listed  in 
Appendix  F.  It  represents  20  major  software  error 
categories  and  164  software  error  subcategories  (Ref[24]:3“3 
thru  3-8)  and  ( Ref [7 ] : 45-50) . 

Seperate  subfiles  were  created  from  the  RADC  files 
for  time-to-fix  and  time-to-error . Time-to-error  was 
computed  as  the  difference  in  time  between  two  successive 
SPRs.  Within  each  SPR,  the  date  the  error  occurred  and  the 
date  it  was  fixed  was  recorded.  From  these  two  dates  the 
time-to-fix  for  each  error  was  computed.  The  beginning  of 
the  test  phase  was  considered  "day  one”  for  computational 
purposes.  All  subfiles  beginning  with  an  ”EF”  prefix 
represent  time-to-error  files  and  all  subfiles  beginning 
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with  an  "FF”  prefix  are  time-to-fix  files  (e.g.  EF1,  EF4A, 
FF1,  FF5G). 

I I . 2 File  characteristics 

11. 2.1  Filel 

Filel  represents  4519  software  errors  collected  by 
TRW  on  a large  command  and  control  software  package  (Project 
3).  The  software  was  written  by  TRW  and  a subcontractor  in 
J0VIAL/J4  and  contained  115,346  source  statements  and  249 
routines.  The  majority  of  errors  were  detected  during 
formal  testing  over  approximately  an  eight  month  period. 
The  command  and  control  software  package  was  tested  in  a 
batch  mode.  The  entire  file  was  used  for  time-to-error  and 
time-to-fix  analysis  and  the  files  were  called  EF1  and  FF1 
respectively  ( Ref [ 24] : 2-2 , 2-3 ) . 

11. 2. 2 File2 

File2  contains  the  operational  data  for  Project  3, 
which  was  referred  to  in  Section  II.2.1.  The  error  data  was 
collected  over  approximately  a nine  month  period.  The 
records  within  this  file  do  not  contain  error  category 
information  and  therefore  time-to-fix  could  not  be  extracted 
by  category.  There  were  451  software  error  records  in  the 
file.  Time-to-error  data  was  extracted  and  called  EF2  (Ref 
[24]:2-2). 


II. 2. 3 File3 

File3  consists  of  records  representing  SPR  data,  from 
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a project  by  TRW  and  a subcontractor  (Project  2).  Again,  it 
was  written  in  J0VIAL/J4  for  a large  command  and  control 
system.  The  software  package  consisted  of  96,931  source 
statements  and  173  routines  and  operated  in  a batch  mode. 
Originally  there  were  436  software  error  records  in  the 
file.  It  consisted  of  two  independent  modifications  and  the 
file  had  to  be  split  into  two  autonomous  subfiles  before 
time-to-error  analysis  could  begin.  The  time-to-error  files 
thus  formed  were  called  EF3A  and  EF3B.  For  time-to-fix  the 
entire  file  was  used  and  was  simply  called  FF3  (Ref  [24]). 

II. 2. 4 File4 

File4  is  a collection  of  2165  software  errors  from  a 
real-time  control  system  for  a land-based  radar  system 
developed  by  Raytheon  Company  over  approximately  a 37  month 
period.  The  majority  of  the  109  software  modules  were 
written  in  J0VIAL/J3,  with  the  remaining  modules  coded  in 
assembly  language.  The  project  was  developed  in  several 
builds.  A build  was  an  independent  computer  development  and 
coding  phase.  Sufficient  data  was  not  recorded  in  the  files 
and  the  necessary  information  was  not  available  from  the 
contractor  on  how  to  break  out  the  file  into  builds.  Again, 
it  must  be  stressed  that  the  data  was  not  collected  with 
this  study  in  mind.  The  time-to-error  data  was  extracted 
and  called  EF4.  Two  means  of  collecting  time-to-fix  data 
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were  used.  One  used  a 365  day  calendar  to  compute  the 
difference  in  time  between  when  the  error  was  found  and  when 
it  was  corrected.  The  file  created  from  this  process  was 
called  FF4A.  Another  file,  FF4B,  was  created  from  fix  data 
that  was  computed  and  recorded  by  the  contractor  using  a 
workday  calendar  (Ref  [29]). 

II. 2. 5 Files 

Files  was  generated  by  Boeing  Aerospace  Company  from 
Software  Problem  Reports  (SPRs)  on  an  avionic  software 
development  package.  There  were  2036  error  reports  over  an 
approximately  27  month  period.  JOVIAL/JB  was  used  and 
accounted  for  approximately  40,000  lines  of  code.  The 
remaining  80,000  lines  of  code  were  written  in  assembly 
language.  The  project  was  developed  using  seven  functions, 
each  function  being  an  entity  of  its  own.  Time-to-error  was 
extracted  for  each  of  these  seven  functions.  The  time-to- 
error  was  computed  by  counting  the  number  of  days  between 
SPRs.  These  seven  files  were  called  EF5A , EF5B,  EF5C,  EF5D, 
EF5E,  EF5F , and  EF5G.  Also,  Boeing  computed  time-to-error 
by  using  a formula  that  totalled  the  number  of  daily  hours 
of  equipment  use  since  the  beginning  of  the  test  phase. 
This  data  was  used  to  extract  the  time  between  SPRs  and  the 
files  EF5AH,  EF5BH,  EF5CH,  EF5DH,  EF5EH,  EF5FH,  and  EF5GH 
were  generated.  Time-to-fix  data  was  extracted  for  the 
entire  file  and  called  FF5.  Boeing  also  computed  time-to- 
fix  by  the  hundredths  of  hour.  This  data  was  extracted  and 
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Chapter  III 
Model  Selection 


1 1 1 . 1 Software  Reliability  Models 

1 1 1 . 1 . 1 Potential  Models 

Numerous  models  have  been  suggested  to 
stochastically  represent  software  reliability.  Listed  at 
the  end  of  this  section  in  Table  II  are  the  models  and  their 
respective  references  that  were  considered  for  possible 
inclusion  in  this  study.  By  no  means  does  the  absence  of  a 
model  from  Table  II  indicate  its  relative  value.  The  models 
listed  are  those  for  which  adequate  literature  could  be 
found . 

Software  reliability  can  be  measured  in  two  ways: 
reliabilty  growth  and  interval  reliability.  The  distinction 
I between  the  two  is  extremely  important.  Reliability  growth 

is  just  what  its  name  indicates,  the  growth  in  reliability 
as  time  increases,  where  time  is  measured  as  starting  on  the 
first  day  of  testing.  It  is  expected  that  as  time  proceeds, 
more  errors  are  removed,  thus  increasing  the  reliability. 
Ideally  all  errors  should  be  found  and  a reliability  equal 
to  1.00  would  be  achieved.  For  interval  reliability,  time 
is  measured  since  the  last  error  and  the  probability  density 
function  represents  the  distribution  of  time-to-error . An 


interval  is  the  time  between  two  successive  errors.  The 
probability  that  an  error  will  be  found  (unreliability) 
increases  as  the  time  since  the  previous  error  increases. 
Thus,  the  reliability  is  high  in  the  beginning.  But,  as  the 
software  package  or  program  is  exposed  longer  and  longer 
since  the  previous  error,  the  probability  of  the  package 
performing  to  specifications  (reliability)  decreases. 
Mathematically  this  can  be  st  '■ed  by 

R(x,T)  = r[X(t)  = 1,  T<t<T+x] 

where  X(t)  = 1 if  the  software  is  operating  at  time  "t”  and 
X(t)  = 0 otherwise.  Inherent  in  this  formulation  is  that  at 
"T”,  the  starting  point  of  the  interval,  the  software  is 
operating  and  the  duration  of  the  interval  is  "x”  (Ref  [1] 
: 8).  Knowledge  of  interval  reliability  would  enable  the 
software  manager  to  ask  what  the  reliability  of  a package 
was  after  being  exposed  to  ”x"  number  of  hours  or  days. 
Reliability  growth  would  enable  him  to  release  a package 
when  it  reached  a predetermined  level  of  reliability.  The 
uses  and  combinations  of  these  two  types  of  reliability  are 
endless.  But  the  crucial  point  being  made  is  their 
difference,  and  that  interval  reliability  and  reliability 
growth  are  two  different  concepts. 

The  software  reliability  models  listed  in  Table  I 
were  based  upon  many  different  principles.  In  order  to 
produce  meaningful  results,  testing  of  the  competing 
software  reliability  models  was  narrowed  to  only  include 
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those  based  upon  interval  reliability.  With  this  constraint 
and  the  lack  of  adequate  data  for  certain  models,  several 
models  were  eliminated  from  inclusion  in  this  thesis,  as 
will  be  discussed  in  the  next  section.  For  further  details 
on  the  models  that  were  eliminated,  references  are  given  in 
Table  I. 


) 
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Table  I 


Potential  Software  Reliability  Models 


Software  Reliability  Model  Reference 


1) 

Shooman’s  Exponential 

Ref([22],t23],[27]) 

2) 

Jel inski -Mo rand a 

De-Eutrophication 

Ref([8],[14],[15],C23] 

3) 

Schick-Wolverton 

Ref([30],[23]) 

4) 

Weiss 

Ref([28]) 

5) 

Corcoran 

Ref([3]) 

6) 

Markovian 

Ref([23],[25]) 

7) 

Nelson 

Ref(t16]) 

8) 

Weibull 

Ref([23] ,[20] ,[27]) 

9) 

Lipow-Schick-Wolverton 

Ref([12],[23]) 

10) 

Lipow-Jelinski-Moranda 

De-Eutrophication 

Ref([12],[23]) 

11) 

Modif ied-Schick-Wolverton 

Ref([12],[23]) 

12) 

Reliabilty  Growth  Modelling 

nef([9],[23]) 

13) 

Mills 

Ref([21],[9]) 

14) 

Je 1 inski -Mo r and a-Geometr ic 
De-Eutrophication 

Ref([14],[15],[23]) 

15) 

Modified  Geometric 

De-Eutrophication 

Ref([23]) 
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III.  1 .2 


Eliminated  Models 


Seven  of  the  models  considered  were  eliminated, 
primarily  because  necessary  data  was  not  contained  within 
the  RADC  files.  A brief  discussion  on  the  reasons  for 
eliminating  each  is  given  below. 

1 1 1 . 1 . 2 . 1 Shopman ' s Exponential 
Software  Reliability  Model 

Unfortunately,  several  data  elements  were  not 
available  for  a complete  testing  of  t-his  model.  The 
processing  rate,  number  of  total  hours  in  test,  number  of 
machine  instructions,  number  of  runs  terminating  in  failure, 
and  total  time  of  successful  and  unsuccessful  runs  were  not 
available  from  the  files.  Rather  than  making  a biased 
estimate  of  these  inputs,  it  was  thought  best  not  to  include 
this  model  in  the  study. 

111. 1.2.2  We i s s Software  Reliability  Model 

The  Weiss  model  required  knowledge  of  the 
probability  of  correcting  an  error  and  the  test  time  for 
successes  and  failures,  neither  of  which  was  available. 

1 1 1 . 1 . 2 . 3 Corcoran  Software  Reliability  Model 

Knowledge  of  N number  of  trials  with  number  of 

failures  and  Nj  number  of  successes  and  the  probability  of 
correction  of  the  error  was  required  for  this  model.  As  in 
the  Weiss  model,  the  information  was  not  available. 


J 


111.1.2.^  Markovian  Software  Reliability  Model 

The  primary  reasons  for  the  elimination  of  this 
•^odel  were  that  it  did  not  fall  into  the  scope  of  the  study 
and  sufficient  time  was  not  available  to  give  a fair 
assessment . 

1 1 1 . 1 . 2 . 5 Nelson  Software  Reliability  Model 

There  was  no  control  over  the  original  test  input 

for  the  modules  from  which  the  data  bases  were  collected. 
Consequently,  the  number  of  inputs  for  which  execution 
failures  occurred  could  not  be  examined  and  this  model  could 
not  be  used. 

111. 1.2.6  Reliability  Growth  Modelling 

As  stated  earlier,  reliability  growth  modelling  is 
concerned  with  the  growth  of  reliability  as  time  increases 
since  the  beginning  of  software  testing.  However,  the 
majority  of  the  models  listed  in  Table  I are  dependent  on 
interval  reliability  and  model  the  distribution  of  time-to- 
error  between  successive  errors.  As  mentioned  earlier,  it 
was  deemed  in  the  best  interest  of  this  study  to  investigate 
only  those  models  based  upon  interval  reliability. 

111. 1.2. 7 Mills  Software  Reliability  Model 

A number  of  known  errors  can  be  seeded  or  tagged 
into  a program,  whose  code  has  been  completed.  On  the  basis 
of  the  number  of  indigenous  and  induced  errors  found  after  a 
period  of  testing,  an  estimate  can  be  computed  for  the 
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initial  error  content  of  the  program.  Seeding  and  tagging 
of  software  modules  were  not  done  for  the  projects  involved, 
thus  eliminating  this  model. 


III. 1.3  Selected  Models 

Eight  models  remain  for  which  adequate  data  exists. 

In  the  subsections  that  follow  a brief  description  is  given 

of  each.  For  each  model  a detection  rate  is  specified, 

where  the  detection  rate  is  defined  as  the  rate  at  which 

errors  are  detected  for  a given  unit  of  time.  Chapter  V 

will  present  further  details  on  the  actual  computation  of 

/ 

the  parameters  for  the  models. 


I II.  1.3.1  Jel inski -Mo rand a 

De-Eutrophication  Model  ( JM) 

In  this  model  it  is  assumed  that  the  initial 
detection  rate  is  given  by  N <t)  , where  N is  the  initial  error 
content,  and  phi  (<t>)  is  the  proportionality  constant 
(Ref[14]:  327).  As  an  error  is  removed,  the  detection  rate 
decreases  in  a stepwise  fashion  as  can  be  seen  in  Figure  2. 


Figure  2.  De-Eutrophication 

Process  Ref  ( [2 1 ] : ^182) 
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Between  errors  the  detection  rate  is  assumed  constant.  The 


resulting  probability  density  function  (pdf)  is 


f(t^)  = <|>[N-(i-1)]EXP{-<t)[N-(i-l)]t^} 


where 

N = initial  error  content 
i = interval 

t.  = the  time  separation  between  the  (i-l)st 
and  "i''th  error 


and 


<j)  = proportionality  constant 


(1) 


Two  basic  assumptions  of  this  model  point  out  potential 
problems  in  using  the  RADC  data  with  this  model.  First,  the 
model  assumes  that  there  is  only  one  error  per  interval. 
However,  the  data  were  collected  such  that  errors  were 
recorded  by  days.  On  any  particular  day  there  could  be  more 
than  one  error.  Jelinski  and  Moranda  recognized  this 

I 

conflict  in  data  they  used  (Ref[8]:  475).  They  proposed  to 
use  each  SPR  date  as  a "stop  date"  and  to  disregard  the 
number  of  errors  occurring  on  this  date.  Regardless  of  how 
many  errors  occurred,  the  software  system  has  failed  to 
function  properly.  This  approach  was  accepted  for  model  JM. 
Another  model  based  on  JM  takes  into  consideration  the 
number  of  errors  per  interval  and  is  discussed  later. 
Secondly,  the  model  assumes  that  each  error  is  corrected  and 
removed  before  the  next  error  occurs.  This  was  not  always 
the  case  as  revealed  by  the  data  in  the  RADC  files,  where  an 
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error  could  be  discovered,  but  other  error(s)  could  have 
been  found  before  the  first  one  was  corrected.  This  dilemma 
could  have  been  caused  by  nonfatal  errors,  which  do  not 
completely  halt  the  software.  A remedy  for  this  problem 
could  not  be  found.  Intervals  could  have  been  based  on 
weeks  instead  of  days,  in  hopes  that  most  errors  were 
corrected  within  the  week.  Inspection  of  the  files  showed 
that  there  still  would  be  overlapping,  and  much  valuable 
information  would  be  lost  by  consolidating  the  errors.  For 
some  files  the  most  common  interval  of  time  between  "stop 
dates”  would  become  one  week.  This  produces  problems  when 
model  parameters  are  estimated.  Taking  all  of  this  into 
consideration,  it  was  felt  that  the  best  approach  was  to 
retain  the  day  as  the  basic  unit  of  time  and  to  use  the 
model  with  the  data  as  is,  even  though  the  errors 
overlapped.  This  approach  does  limit  the  validity  of  the 
results  produced.  Only  with  further  error  collection  and 
actual  model  testing  will  the  full  implication  be  realized. 

I I I . 1 . 3 . 2 Schick-Wolverton  Model  (S W) 

This  model  is  quite  similar  to  JM , but  the  detection 
rate  is  dependent  on  the  time  interval  between  successive 
errors,  as  well  as  N and  phi.  The  resulting  pdf  is 


f(t^)  = «j>[N-(i-1  )]t.EXP{-(4>/2)[N-(i-1  )]t?}  (2) 

where 

N = initial  error  content 
i = interval 


J 
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t.  = the  time  separation  between  the  (i-l)st 
^ and  the  "i"th  error 


and 


<t>  . = proportionality  constant 


The  input  data  were  used  as  "stop  dates",  disregarding  the 
number  of  errors  at  each  stop.  Also,  the  data  were 
accepted,  realizing  that  the  errors  do  overlap. 


1 1 1 . 1 . 3 . 3 We  i_bin  1 1_  Model  UM  1_) 

This  model  was  applied  to  software  reliability  by 
Wagoner  (Ref  [27]:D-1).  Wagoner  used  the  least  square 
method  for  estimating  the  scale  and  shape  parameters,  but 
for  this  thesis  parameter  estimation  was  performed 
differently  as  will  be  presented  in  Chapter  V.  The  pdf 
is : 


f(t.  ) 
1 


a//3  (t^/i3)“"^EXP[-(t. //3)“  ] 


r 

t.  > 0 
1 — 


/3  > 0 


0 elsewhere 


a > 0 


where 


V. 


t.  = the  time  separation  between  the  (i-l)st 
^ and  "i"th  error 

0 - scale  parameter 

a = shape  parameter 

and 


(3) 


i = interval 
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Again,  inputs  are  treated  as  "stop  dates",  disregarding  the 
number  of  errors  occurring  at  each  interval. 


111.1.3.^  Jel inski -Mo rand  a Geometric 

De- Eutrophication  Model  (~JMG) 

This  model  is  similar  to  JM , but  the  detection  rate 
decreases  in  a geometric  progression  on  the  occurrence  of 
each  individual  error  (Ref  [14]:327).  The  initial  error 
content  is  no  longer  fixed  at  N,  as  in  JM . An  infinite 
number  of  errors  can  occur  over  the  life  of  the  software 
package.  Only  one  error  can  occur  per  interval  of  time, 
thus  restricting  the  data  to  be  used  as  "stop  dates"  as 
before.  The  pdf  for  this  model  is: 


f(t^)  = Dk^"''EXP(-  Dk^"‘'t^) 


(^1) 


where 

i = interval 

t.  = the  time  separation  between  the  (i-l)st 
and  "i"th  error 

D = initial  detection  rate 

k = ratio  constant 


I I I . 1 . 3 . 5 Li pow-Jelinski-Moranda 
De-£'u trophication  TLJM) 

Lipow  (Ref  [12]: 1-7)  has  suggested  that  numerous 
errors  in  an  interval  can  be  accounted  for  by  making 
modifications  to  the  detection  rate  of  the  JM  model.  The  new 
detection  rate  incorporates  the  number  of  errors  found  up 
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through  the  (i-l)st  interval.  Throughout  the  parameter 
estimation  process  the  quantity  (i-1),  that  was  used  in  the 
JM  model,  is  replaced  by  (n^_^),  which  is  the  cumulative 
number  of  errors  found  up  through  the  (i-l)st  interval.  The 
resulting  pdf  becomes 


f(t^)  = «j>(N-n^_^)EXP[-<J>(N-n._^)t^]  (5) 

initial  error  content 
interval 

the  time  separation  between  the  (i-l)st 
and  ”i"th  error 

cumulative  number  of  errors  found 
through  the  (i-l)th  time  interval 

proportionality  constant 

I I I . 1 . 3 . 6 Lipow-Schick-Wolverton  Model  (LSW) 

In  order  to  account  for  any  number  of  errors  in  an 
interval  in  the  SW,  Lipow  (Ref  [11]: 1-7)  modified  the  SW  in 
the  same  manner  suggested  for  the  JM . The  pdf  becomes 

f(t^)  = <t)(N-n._^)t.EXP  [-(<t>/2)(N-n._^)t|]  (6) 

where 

N = initial  error  content 
i = interval 

n.  . = cumulative  number  of  errors  found  through 
the  (i-l)th  time  interval 


where 

N = 

i = 

t.  = 
1 

"i-1 

and 

4>  = 
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and 


t.  = the  time  separation  between  the  (i-l)st 
^ and  "i"th  error 


<j>  r proportionality  constant. 


I I I . 1 . 3 . 7 Modified  Lipow-Schick- 
Wq Iverton  Model  CMLSW) 

Another  variation  of  SW  and  its  detection 

presented  by  Lipow  (Ref  [12]: 1-7).  The  pdf  for 


rate  was 
the  model 


is : 


f(t.)  = «|)(N-n._^)[T._^  + (t./2)]EXP{-4>(N-n._^)[T.__^  + (t^/2)]t.  } 
where  (7) 

N = initial  error  content 
i = interval 

n.  = cumulative  number  of  errors  found  through 
the  (i-l)th  time  interval 

t.  = the  time  separation  between  the  (i-l)st 
^ and  "i"th  error  ' 

T = cumulative  time  through  the  (i-l)st  error 
i-1 

and 

(|>  = proportionality  constant 

I 

.111.1.3.8  Weibull2  Model  (WM2) 

WM2  is  identical  to  WM1,  except  that  it  accounts  for 
the  number  of  errors  in  interval  ”i'*.  In  order  to  account 
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for  the  number  of  errors  in  each  time  interval,  each 
interval  is  replicated  for  each  error  in  the  interval.  For 
instance,  an  interval  that  was  two  days  long  and  contained 
five  errors  would  be  represented  by  five  data  points,  where 
each  data  point  has  an  interval  of  two  days.  Each  error  is 
treated  as  a separate  entity,  independent  of  the  other 
errors  in  the  same  or  another  interval,  and  represents  a 
data  point.  The  pdf  is  the  same  as  Eq.(3),  but  there  are 
many  more  data  points  considered  in  the  parameter  estimation 
process . 

I I I . 1 . 3 . 9 Modified  Geometric 

De- Eutrophication  Model  (MG) 

Sukert  (Ref  [23]:  19-21)  modified  the  JMG  by 

allowing  for  more  than  one  error  per  interval.  It  was  based 

on  Lipow's  formulas.  The  pdf  is: 

f(t.)  = D(k"i-'')EXP[-  D(k"i‘‘')t.]  (8) 

1 1 

where 

i = interval 

D = initial  detection  rate 
k = ratio  constant 

n.  = cumulative  number  of  errors  found 
through  interval  "i-1" 

t.  = the  time  separation  between  the  (i-l)st 
^ and  "i”th  error 
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III.  1 .4 


Software  Reliability  Model  Summary 
With  the  addition  of  the  extra  Weibull  model,  nine 
software  reliability  models  were  chosen  for  model 
validation . 

The  first  four  models  only  required  "tj^"  and  "i”  for 
estimation  of  their  parameters,  treating  "t^"  as  the  time 
separation  between  "stop  dates"  (SPR  dates) . These  models 
will  henceforth  be  classified  as  Group  I models.  The  last 
five  models  account  for  the  number  of  errors  per  SPR  date 
and  will  be  classified  as  Group  II  models.  They  require 
"t^","i",  and  the  number  of  errors  occurring  per  SPR  date. 
For  these  models,  "t^"  is  the  time  separation  between 
successive  SPRs.  Table  II  lists  the  nine  models,  their 


acronyms,  and  the  parameters  requiring  estimation. 

It  should  be  noted  that  one  additional  model  was 
found  late  in  the  development  of  this  thesis^  that  could 
possibly  be  included  in  Group  II  of  Table  II.  It  is  the 


Geometric  Poisson  Model  bv  Moranda  (Ref  [14]:R10).  There 
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Table  II 

Selected  Software  Reliability  Models 


Group  I 

Input  Requirements  - Length  of  each  interval 
(number  of  days  between  each  pair  of  successive  errors) 


Model  name 

Model  acronym 

Parameters 

Jelinski-Moranda 

De-Eutrophication 

JM 

N and  phi 

Schick-Wolverton 

sw 

N and  phi 

Weibulll 

WM1 

a and  ^ 

Je 1 inski -Mo r and a-Geometric 
De-Eutrophication 

JMG 

D and  k 

Group 

Input  Requirements  - Length  of  i 
of  errors  within 

II 

each  interval  and 
each  interval 

the  number 

Model  name 

Model  acronym 

Parameters 

Li pow-Jelinski-Moranda 
De-Eutrophication 

LJM 

N and  phi 

Li pow-Schick-Wolverton 

LSW 

N and  phi 

Modif ied-Lipow-Schick- 
Wolverton 

MLSW 

N and  phi 

Weibull2 

WM2 

a and  p 

Modified 

Geometric  De-Eutrophication 

MG 

D and  k 

29 


III. 2 Tirne-to-Fix 


111. 2.1  Background 

If  within  each  software  error  category  or 
subcategory  the  random  variable  time-to-fix  could  be 
modelled,  a software  manager  could  determine  the  worst  case 
estimate  on  how  long  it  would  take  to  fix  a particular 
error.  Of  course,  this  is  assuming  that  the  type  of  error 
is  known,  which  is  not  always  the  case.  Ideally,  a software 
manager  could  use  established  pdf's  for  error  categories, 
determine  the  worst  case  estimate,  and  thereby,  eliminate 
the  haphazard  guesswork,  which  so  often  underestimates  fix 
time  and  results  in  cost  overrun  and  strained  working 
relationships . 

111. 2. 2 Time-To-Fix  Program  Selection 

Time-to-fix  is  the  difference  in  time  between  when 

the  error  was  discovered  and  when  it  was  corrected.  It 
could  be  measured  in  central  processing  time,  days,  weeks, 
or  any  of  various  ways.  The  data  in  the  RADC  files  was 
recorded  by  days.  No  information  could  be  found  on  other 
previous  ettempts  to  model  time-to-fix.  A computer  program 
(Program  Model)  written  by  'T.  L.  Regulinski  (Ref  [18])  was 
used  as  the  basis  for  determining  the  pdf  that  governed  the 
time-to-fix  data.  Program  Model  accepts  time-to-fix  data 
and  estimates  the  parameters  for  the  probability  density 
functions  listed  in  Table  III.  After  parameter  estimation. 
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each  pdf  is  tested  for  goodness  of  fit  by  the  Kolmogorov- 


Smirnov  Test  (Ref  [5] , C 1 0] , [ 1 33 ) • Each  distribution  is 
either  accepted  or  rejected  at  a significance  level  of  0.20. 
Of  those  accepted,  the  Likelihood  Ratio  Test  (Ref  [4]  and 
[19])  is  used  to  select  the  one  that  best  models  the 
particular  time-to-fix  data  set.  A modification  to  Program 
Model  allowed  a data  file  containing  two  or  more  different 
error  categories  to  be  sorted  by  error  category  and 
subsequently  have  a distribution  selected  for  time-to-fix 
within  each  category.  Henceforth,  this  new  program  will  be 
called  Program  Time-To-Fix. 
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Table  III 


Time-To-Fix  Density  Functions  (Ref  [18]) 
Distribution  Name 

1)  Uniform 

2)  Exponential 

3)  Weibull 

4)  Normal 

5)  Log-normal 

6)  Gamma 

7)  Logistic 

8)  Extreme  Value-Largest 

9)  Extreme  Value-Smallest 

10)  Pareto 


Chapter  IV 


Data  Preparation 

IV. 1 Preparation  Background 

Six  files  were  received  from  the  RADC  repository. 
Separate  computer  programs  were  written  to  extract  the  data 
from  the  RADC  files  because  each  contractor  recorded  the 
data  in  different  formats.  The  files  contained  virtually 
every  category  in  Appendix  F.  However,  not  all  of  the 
categories  were  considered  legitimate  software  errors  in 
respect  to  this  study.  Categories  "L”,  "T”,  "U”,  "V”,  and 

"X"  were  deemed  irrelevant  for  this  study  and  were  deleted 
as  the  time-to-error  and  time-to-fix  files  were  being 
created.  Respectively  these  categories  are  user  requested 
changes,  operator  errors,  questions,  hardware  er^'ors,  and 
non-reproducible  errors.  None  of  these  are  true  software 
errors,  even  though  they  definitely  do  impact  the 
development  of  a software  package.  Also,  many  records  in 
Filel,  File2,  File3  and  File4  were  classified  as  explanatory 
software  problem  reports  (SPRs).  These  records  were  not 
representative  of  an  error  in  the  executable  code  and 
therefore,  were  not  included  in  the  time-to-error  and  time- 
to-fix  files. 
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IV. 2 


Time-To-Er ror  Data 


Three  approaches  could  be  taken  for  extraction  of 
time-to-error  data  for  the  nine  software  reliability  models. 

Option  1)  Treat  each  file  as  a whole 

and  extract  the  time-to-error 
data  in  a chronological  order. 

OR 

Option  2)  Determine  if  the  files  were  made 

up  of  separate  software  development 
entities,  independent  of  others  in 
the  file,  and  then  create  time-to-error 
data  in  a chronological  order  for 
each  subfile. 

OR 

Option  3)  Isolate  every  program  in  each  file 
and  create  a chronological 
time-to-error  subfile  for  each 
program . 

Option  number  three  was  discarded,  because  in  a 
large  system,  any  particular  program  may  sit  idle  for  a long 
period  of  time.  Time  between  successive  errors  would  be 
meaningless  unless  this  idle  time  could  be  accounted  for. 
Idle  times  for  particular  programs  could  not  be  obtained 
from  the  contractors.  Also,  much  data  could  not  be  used, 
because  many  programs  only  had  one  or  two  errors  recorded. 
Modelling  with  such  a few  number  of  data  points  would 
produce  results  of  questionable  value. 

A compromise  between  Option  1 and  Option  2 was 
required.  Filel,  File2,  and  File4  could  not  be  split  into 
separate  entities  and  were  used  in  their  entirety,  with 
time-to-error  being  computed  chronologically  for  the  entire 


FileM  had  numerous  records  which  did  not  contain  the 

date  the  error  was  found.  As  recommended  by  RADC  and  the 

contractor,  each  of  these  records  arbitrarily  had  the  date 

that  the  error  was  found  set  equal  to  the  date  that  the 

error  was  fixed.  Option  2 was  used  in  the  creation  of 

subfiles  for  File3.  During  the  creation  of  the  software 

package  from  which  the  error  data  for  FileS  was  collected, 

two  independent  software  subprojects,  called  "modifications" 

by  the  contractor,  were  implemented.  For  each  of  these 

"modifications"  a time-to-error  file  was  created,  each 

having  its  time-to-error  computed  independently  from  the 

other's.  Similarly,  File5  had  to  be  split  into  seven 

different  subfiles,  each  representing  time-to-error  data  for 

a particular  functional  group  of  software.  Each  functional 

group  was  independently  developed  to  fulfill  a particualar 

requirement  for  the  master  avionics  system.  Each  group  had 

its  own  software  and  a time  schedule  that  was  unique.  All 

the  files  created  thus  far  have  had  time-to-error  computed 

as  the  difference  in  days  between  two  consecutive  errors. 

In  addition,  File5  contained  time-to-error  recorded  and 

( 

calculated  by  the  contractor  for  the  hourly  equipment  use 
time  (Ref  [7]:20).  This  data  was  split  into  seven  separate 
subfiles,  each  containing  time-to-error  data  for  a specific 
functional  group.  A few  records  within  four  of  these 
functions  had  the  hourly  data  recorded  incorrectly  by  the 
contractor.  When  the  time  between  consecutive  errors  was 
computed  it  was  not  always  a positive  value.  This  could  not 


a linear 


be  possible.  To  correct  these  few  records 
interpolation  estimate  between  the  SPR  dates  was  used  to 
estimate  the  number  of  hours  between  errors. 

Since  File2  is  the  operational  data  for  the  same 
project  as  that  used  in  the  creation  of  Filel,  it  seemed 
quite  reasonable  to  track  the  entire  project  and  to  create  a 
file  which  contained  data  from  Filel  (test  data)  and  File2 
(operational  data).  File2  was  concatenated  with  Filel  and 
the  resulting  file  was  then  sorted  by  SPR  date,  thus 
creating  error  data  for  a continuing  project. 

Twenty  time-to-error  files  resulted  from  the  above 
procedures  and  are  listed,  along  with  a brief  description, 
in  Table  IV.  No  mention  was  made  of  File6,  because  it  did 
not  contain  data  on  when  errors  were  found.  File  EF5C  is 
given  in  its  entirety  in  Appendix  B as  an  example  of  the 
data  created. 


Table  IV 


1 

I i 

i 

■A 


Time-To-Error  Input  File  Table* 


RADC 

File 

Extracted 

Time-To-Error 

File 

Description 

Filel 

EF1 

Contains  152  records 

from  Filel 

File2 

EF2 

Contains  86  records 

from  File2 

File1/2 

EF1/2 

Contains  238  records 
concatenation  of  File2 

from  the 
! onto  Filel 

Files 

EF3A 

Contains  54  records 

of  Files 

from  MODA 

EF3B 

Contains  30  records 

of  Files 

from  MODB 

File4 

EFi| 

Contains  435  records 

from  File4 

File5 

EF5A 

Contains  149  records 
build  "A”  of  Fiie5 
in  days 

from 

as  computed 

EF5B 

Contains  146  records 
build  "B"  of  File5 
in  days 

from 

as  computed 

EF5C 

Contains  51  records 

build  "C"  of  File5 
in  days 

from 

as  computed 

EF5D 

Contains  83  records 

build  "D"  of  File5 
in  days 

from 

as  computed 

EF5E 

Contains  92  records 

build  "E"  of  File5 
in  days 

from 

as  computed 

EF5F 

Contains  102  records 
build  ”F''  of  File5 
in  days 

from 

as  computed 

EF5G 

Contains  80  records 

build  "G"  of  File5 
in  days 

from 

as  computed 

* All  time-to-error  files  do  not  represent  records  with 

error 

categories  "L", 

"T",  "U",  "V",  and  "X” 

or  those 

that 

are  explanatory 

in  nature. 

t 

t 
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! Table  IV  (Continued) 

Time-To-Er ror  Input  File  Table* 


RADC 

File 

Extracted 

Time-To-Error 

File 

Description 

File5 

EF5AH 

Contains  131  records 
build  "A"  of  Files 
in  hours 

from 

as  computed 

EF5BH 

Contains  I30  records 
build  ”B”  of  Files 
in  hours 

from 

as  computed 

EF5CH 

Contains  M9  records 

build  "C  of  Files 
in  hours 

from 

as  computed 

EF5DH 

Contains  7S  records 

build  "D"  of  Files 
in  hours 

from 

as  computed 

EF5EH 

Contains  109  records 
build  ”E"  of  Files 
in  hours 

from 

as  computed 

EF5FH 

Contains  90  records 

build  "F"  of  Files 
in  hours 

from 

as  computed 

EF5GH 

Contains  76  records 

from 

build  "G”  of  File5  as  computed 
in  hours 


* All  time-to-error  files  do  not  represent  records  with 
error  categories  "L",  "T" , "U",  "V",  and  "X”  or  those 
that  are  explanatory  in  nature. 
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IV. 3 Time-To-Fix  Data 

Splitting  the  files,  as  was  done  for  time-to-error 
data  files,  was  not  necessary  for  the  creation  of  time-to- 
fix  data  files.  Time-to-fix  is  unlike  time-to-error  in  that 
all  of  the  information  required  for  determining  time-to-fix 
is  recorded  in  one  record  and  requires  no  knowledge  of  the 
previous  record.  Each  contractor  had  his  own  software 
development  techniques.  For  a particular  contractor  the 
techniques  remained  the  same  from  one  "build'*, 
"modification",  or  "functional  grouping"  to  the  next.  Since 
the  same  techniques  were  used  within  any  particular  software 
project  it  appeared  unnecessary  to  split  the  files  by 
"modifications",  "builds",  or  "functional  groups"  for 
creation  of  the  time-to-fix  data  files.  Combining  all  of 
the  files  into  one  large  time-to-fix  file  would  be  quite 
unreasonable  because  each  contractor  developed  his  software 
using  different  techniques.  Besides,  one  of  the  reasons  for 
modelling  time-to-fix  was  to  see  if  there  existed 
distributions  which  were  the  same  from  one  project  to  the 
next . 

A separate  time-to-fix  file  for  Filel,  File2,  File3, 
File4,  and  File5  was  created,  and  each  record  within  the  new 
files  consisted  of  the  number  of  days  it  took  to  fix  the 
error  and  the  error  category  from  Appendix  F.  Any  error 
found  and  fixed  on  the  same  day  was  considered  to  have  a fix 
time  of  0.5  days. 
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File4  also  had  time-to-fix  computed  using  a work 
calendar.  The  difference  between  when  the  error  was  found 
and  when  the  error  was  fixed  did  not  include  weekends  or 
holidays.  This  data  was  extracted  and  used  to  create  file 
FF4B. 

In  File5  additional  data  was  available,  so  that  a 
file  (FF5H)  could  be  created  where  time-to-fix  was 
represented  by  the  hundredths  of  hours  to  fix  (Ref  [73:20). 

Six  files  resulted  from  the  above  procedures  and  are 
listed  along  with  a brief  description  in  Table  V. 
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rf 

1 

I 

I 

Table 

V 

f- 

i 

I 

Time-To-Fix  Input 

File  Table* 

i 

RADC 

File 

Extracted 

Time-To-Fix 

File 

Description 

Filel 

FF1 

Contains 

using 

2172  records  from  Filel 
a 365  day  calendar 

File3 

FF3 

Contains 

using 

219  records  from  File3 
a 365  day  calendar 

File4 

FF4A 

Contains 

using 

1254  records  from  File4 
a 365  day  calendar 

FF4B 

Contains 

using 

1254  records  from  File4 
a workday  calendar 

Files 

FF5 

Contains 

using 

1603  records  from  FileS 
a 365  day  calendar 

FF5H 

Contains 

1603  records  from  FileS 

as  recorded  in  hundredths 
of  the  hour 


* All  time-to-fix  files  do  not  represent  records  with 
error  categories  ”L" , "T",  "U",  "V",  and  "X”  or 
those  that  are  explanatory  in  nature. 


IV. 4 Computation  of  Elapsed  Time 

Except  for  the  error  and  fix  files  that  have  a 
suffix  of  ”H",  the  calendar  day  is  the  basic  unit  of  time. 
Each  SPR  is  represented  by  a record  and  within  each  record 

I 

all  dates  were  recorded  by  month/day/year. 

A seven  year  calendar  was  established  to  convert 
each  month/day/year  entry  in  a record  into  the  number  of 
days  since  1 January  1970.  The  date  1 January  1970  was 
selected  because  it  preceeded  the  earliest  SPR  and 
computations  could  be  easily  made  using  it.  Once  each  time- 
to-error  or  time-to-fix  date  was  transformed  into  the  number 
of  days  since  1 January  1970,  the  number  of  days  between  the 
time-to-error  or  time-to-fix  occurrences  were  easily 
calculated  by  subtracting  the  latest  date  from  the  earliest 
date . 

A possible  legitimate  criticism  of  using  the  day  as 
the  unit  of  measure  would  be  if  the  software  package  sat 
idle  a lot  or  consisted  of  one  program  which  was  used 
sporadically.  But  on  a large  system,  which  is  in  continuous 
use,  the  day  may  be  an  appropriate  unit  of  measure.  CPU 
time  has  been  suggested  as  an  alternative.  In  the  actual 
operation  of  a large  computer  system  or  a real-time  system, 
it  is  sometimes  difficult  to  track  CPU  utilization 
accurately.  Data  were  not  recorded  in  CPU  time  for  the  RADC 
files,  but  in  future  error  collection  efforts  CPU  time  will 
undoubtedly  be  collected. 


Four  different  contractors  built  the  files  and  each 
file  was  recorded  in  a different  format.  Therefore, 
numerous  computer  programs  were  written  to  delete,  extract, 
sort,  and  merge  the  records  into  identical  formats  which 
were  usable  by  a time-to-error  program  and  time-to-fix 
program . 


IV. 5 File  Limitations 

The  creation  of  the  files  in  Table  IV  was  not  a "cut 
and  dry"  decision.  For  instance,  File4  was  collected  from  a 
project  which  consisted  of  several  independent  software 
"builds".  Each  "build"  was  developed  on  a unique  time 
schedule  and  ideally  each  "build"  should  have  its  own  time- 
to-error  subfile,  but  adequate  information  was  not  available 
in  the  file  or  from  the  contractor  to  permit  segregation  by 
"build". 

The  start  date  had  to  be  determined  for  each  time- 
to-error  file.  For  files  EF1,  EF2,  EF1/2,  and  a few  of  the 
subfiles  in  File5  the  actual  date  at  which  formal  testing 
began  was  specified  in  the  contractor's  reports.  On  other 
files  an  official  date  at  which  formal  testing  began  ("day 
one")  could  not  be  obtained.  Only  the  detection  date  of  the 
first  error  was  given  in  these  cases.  A date  had  to  be 
established  and  three  alternatives  seemed  most  logical: 

Alternative  1)  Use  the  date  of  the  first 
SPR  as  "day  one" 

Alternative  2)  Use  the  date  of  the  first 

SPR  minus  one  day  as  "day  one" 
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Alternative  3)  Plot  the  data  points  and  fit 

a straight  line  to  the  points. 
Project  this  line  backwards  to 
a date  that  would  correspond 
to  "day  one". 

Any  of  the  three  alternatives  would  be  an  approximation  of 
the  true  "day  one".  If  a software  package  is  being  tested 
thoroughly  it  is  not  unlikely  that  the  first  error  will  be 
found  within  the  first  day  of  testing.  Therefore,  "day  one" 
was  computed  using  Alternative  2 for  those  files  requiring 
estimation  of  "day  one".  Declaring  the  date  of  the  first 
SPR  as  "day  one"  as  done  in  Alternative  1 would  lose 
valuable  data  on  the  first  SPR.  Actual  usage  of 
Alternative  3 projected  "day  one"  back  to  a date  which  was 
unacceptable.  It  was  felt  that  the  data  from  the  first  SPR 
was  too  important,  especially  for  those  models  requiring  the 
total  number  of  errors  per  interval,  to  be  lost  by  using 
Alternative  1 and  Alternative  3 gave  unreasonable 
predictions  of  "day  one".  Alternative  1 was  tried  for 
several  files  resulting  in  estimated  parameters  that  were 
only  slightly  different  by  no  more  than  a magnitude  of  one 
and  distributions  that  were  the  same  as  for  those  using 


Chapter  V 


Parameter  Estimation  and  Test  of  Hypothesis 


V. 1 Introduction 

A means  of  parameter  estimation  and  discrimination 
between  the  nine  selected  software  reliability  models  had  to 
be  established.  These  functions  were  incorporated  into  a 
computer  program  called  Program  SRHOD  (See  Apppendix  A). 

V . 2 Parameter  Estimation 

Each  software  reliability  model  has  certain 
parameters  that  must  be  estimated  (See  Table  II).  The 
models  in  Group  I of  Table  II  required  only  the  time-to- 
error  ( t^ ) data  from  the  data  base,  while  those  in  Group  II 
required  time-to-error  and  the  total,  number  of  errors  per 
interval  from  the  data  base.  Both  of  these  inputs  were 
available  from  the  RADC  files.  From  the  extracted  raw  data, 
which  consisted  only  of  the  time  between  successive  SPRs  and 
the  number  of  errors  within  the  interval,  numerous 
calculations  had  to  be  made  before  parameter  estimation 
could  proceed.  The  cumulative  number  of  errors  up  through 
each  interval,  the  cumulative  time  up  through  each  interval, 
the  total  number  of  time  intervals,  and  the  total  number  of 
errors  had  to  be  computed.  Various  other  summations  were 
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necessary  before  parameter  estimation  could  be  completed. 
These  summations  can  be  found  in  the  equations  that  follow. 

To  estimate  N and  phi  for  the  JM , SW,  LJM,  LSW,  and 
MLSW  models  the  following  formulas  (Ref  [12]:2-3)  were  used: 
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M 

J 

K 


= length  of  time  interval  in  which 

”M  '•  errors  were  observed 
1 

= cumulative  time  through  (i-l)st  interval 


i-1 

^i-1  = 


= 0 


= cumulative  number  of  errors  observed 
up  through  the  (i-l)st  time  interval 


\ 


n 


i-1 


i-1 
I M 
r=1 


r 


= number  of  errors  in  interval  ”1'' 
= total  number  of  time  intervals 
= total  number  of  errors  observed 


" =i"i 

1=1 

The  formulas  are  a consolidated  version  of  several 
equations,  each  used  to  estimate  N and  phi  for  a particular 
model,  utilizing  the  maximum  likelihood  function.  They  are 
basically  the  same  as  stated  by  Lipow  (Ref  [11]:2-3)  with 
the  exception  that  equations  9 and  10  also  consider 
estimation  of  parameters  for  models  JM , SW,  and  LSW.  In  all 
cases  the  maximum  likelihood  estimate  for  N and  phi  were 
used  as  explained  in  (Ref  [8 ] , [ 1 1 ] , [ 1 4]  ,[15], [231). 

Estimation  of  alpha  and  beta  for  WM1  and  WM2  used 
formulas  taken  from  Program  Model  (Ref  [18]  and  [20]). 


» .6079271 

(11) 

^ = EXP(R/J  + .5772/ a ) (12) 

where 

j = total  number  of  time  intervals 
R = J]^log(t^  ) 

S = y log(t2) 
i=1  ^ 

t^  = length  of  time  interval 

The  two  geometric  models,  JMG  and  MG,  required  "D” 
and  '•!<"  to  be  estimated.  Given  that  the  pdf  for  the  JMG  is. 

f(t  ) = Dk^“^  EXPC-Dk^'Ui  ) (13) 

where  ^ 

i = interval 

t^  = length  of  time  interval 
D = initial  detection  rate 
k = ratio  constant 


the  maximum  likelihood  estimates  for  *'D”  and  "k”  can  be 
determined  (Ref  [14]:330)  and  (Ref  [11]:1).  If  given  a 

sample  of  time-to-error s (t^,t2, tj)  the  likelihood 

function,  denoted  by  "L",  would  be 


8 


'•  ’WP" 


L(ti,t2, ,tj)  = ["[oki"'' EXP(-Dki-1ti  ) 

i = 1 


(14) 


where  ”j"  equals  the  number  of  intervals.  Taking  the 
natural  logarithm  of  both  sides  of  equation  (14)  produces 


j j • 1 

log  L = [j  logD]  + [ f (i-Dlogk]  - [ 7 Dk^-U.  ] (15) 

i^1  i^1  ^ 


Maximizing  equation  (15)  with  respect  to  ”D"  yields 


61ogL  j j . , 

= — - 7 ki-1  ti  = 0 

&D  D i=1 


and  maximizing  with  respect  to  "k"  yields 


(16) 


61ogL 

6k 


J J . 

7 (i-1 ) - D 7 (i-1  )ki-2  t;  = 0 
it*!  i4'1  ^ 


(17) 


The  solution  of  equation  (16)  for  "D"  can  then  be 
substituted  into  equation  (17)  to  arrive  at 


7 ik^  ti 
i^l 


I 

iTI  ^ 


j + 1 


(18) 
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Since  "j”  and  the  ' s are  known  quantities,  equation  (18) 
can  be  solved  for  "k”.  The  solution  will  be  called  "(<"  and 
will  be  a maximum  likelihood  estimate  for  "k".  Likewise, 
the  "It”  can  be  substituted  into  equation  (16)  to  form 


i 

4 

i 

I 

! 

j 

1 

1 


A 

D 


j 


(19) 


from  which  a maximum  likelihood  estimate  for  ”D"  can  be 
derived . 

Similiarly  the  maximum  likelihood  estimates  of  ”k” 
and  "D"  for  MG  were  derived  to  yield 


i/l  i-Utl  ^ 


-I 


iTI 


where 


i = interval 


j = total  number  of  time  intervals 


n 


i-1 


cumulative  number  of  errors  up  through 
the  (i-l)st  time  interval 


(20) 


(21) 
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.1 
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V. 3 Model  Restrictions 

One  problem,  as  recognized  by  Lapadula  (Ref  [91) 
and  Sukert  (Ref  [23]),  did  arise  in  all  models  requiring  an 
estimation  of  "N”.  To  solve  equation  (9)  for  ''N”  (estimate 
for  the  initial  error  content),  incremental  values  were 
substituted  for  ”N”  until  a value  was  found  that  would  make 
the  left  side  of  equation  (9)  equal  to  its  right  side. 

For  several  data  sets  Program  SRMOD  aborted  because 
the  computer  time  spent  trying  to  find  such  an  equality 
exceeded  the  allowable  execution  time  permitted  on  the 
processing  computer.  In  all  cases  the  estimate  for  "N”  at 
the  time  of  abortion  was  far  greater  than  would  be  expected. 
If  it  is  assumed  the  models  are  valid,  then  the  software 
projects  from  which  the  data  came  from  would  have  had 
serious  problems.  No  information  could  be  found  that  would 
indicate  this  fact.  No  positive  reason  could  be  found  that 
would  explain  why  some  data  files  had  this  problem  and 
others  not. 

Sukert  (Ref  [231:41)  suggested  that  it  could  be 
caused  by  short  time  intervals  consisting  of  many  errors  at 
the  end  of  a file.  This  pattern  was  prevalent  in  those 
files  which  had  the  problem  in  this  study. 

The  simplest  and  most  convenient  means  found  to 
alleviate  this  problem,  so  that  testing  could  proceed 
without  being  aborted,  was  to  allow  ”N”  (the  estimate  for 
the  initial  error  content  of  the  software  package)  to  reach 
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n (the  actual  number  of  errors  found  thus  far  in  the 
software  package)  plus  10,000.  At  this  point  it  was  deemed 
highly  unlikely  that  a reasonable  solution  for  "N”  could  be 
determined.  When  this  situation  occurred,  "N”  was  set  to 
this  arbitrarily  large  sum.  After  all  restrictions  were 
resolved  and  the  parameters  were  estimated  for  the  models,  a 
test  of  hypothesis  was  selected  to  discriminate  between  the 
models . 


V.4 


Likelihood  Ratio  Test 


The  Likelihood  Ratio  Test  (LRT)  was  chosen  as  the 
easiest  means  of  discriminating  between  the  nine  different 
distributions.  The  LRT  is  defined  by 


where 


L(T)  = 


f^(t!H^) 

f (t|H^) 
0 0 


(22) 


Hq:  ■- 


G(t) 


F^(t! e^) 


The  null  hypothesis,  H^,  is  that  the  sample  of  taken 
from  a population  with  an  unknown  cumulative  distribution 
function,  G(t),  is  equal  to  a theoretical  distribution, 
Fo(tleo),  where  Bq  is  the  known  parameter(s)  of  "F".  The 
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null  hypothesis  can  be  tested  against  the  alternative 
hypothesis,  which  states  that  G(t)  equals  another 
theoretical  distribution,  F^(t|ei).  Given  that  the  t^'s  are 
statiscally  independent  then  the  quantity  f^(T|H^)  equals 
the  joint  pdf  of  the  t^’s.  The  criterion  against  which  L(T) 
is  measured  for  acceptance  or  rejection  of  a given 
hypothesis  is  called  the  threshold  of  the  test  and  is 
denoted  by  "T'.  The  threshold  is  determined  by  the  cost  of 
making  a correct  or  incorrect  decision  based  upon  apriori 
probabilities  of  the  hypotheses  in  question.  For  this 
application  it  was  assumed  that  the  distributions  under  test 
had  equal  likelihood  of  occurrence  and  that  the  cost  for  an 
incorrect  decision,  regardless  of  the  decision,  was  the 
same.  Also,  it  was  assumed  there  was  no  penalty  for  a 
correct  decision.  Given  these  assumptions  "r|''  equals  one 
and  the  LRT  can  be  written 


L(T) 


>Ht  >H 

71  or  further  ln[L(T)]  ^ ln(1.)  (23) 


<H 


0 


<H. 


which  states  that  if  the  log  of  L(T)  is  less  than  the  log  of 
one,  choose  Hg,  if  it  is  greater  than  the  log  of  one,  choose 
H-].  The  computational  algorithm  utilized  in  Program  SRMOD 
was  based  on 


2;{ln[f^(ti  - j;{ln[fQ(ti  !Hq)]} 


i=1 


i = 1 


>H, 


<H 


(2i<) 
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This  approach  can  be  repetitively  applied  to  test 
several  hypothesis  against  one  another.  Eventually,  one 
hypothesis  will  be  selected  over  all  others.  As  an  example, 
two  models  from  Group  II  are  tested  against  each  other  using 
EF1  as  the  sample  population.  The  hypotheses  are 


: Distribution  is  the  Modified  Geometric  Model 
: Distribution  is  the  Lipow-Schick  Wolverton  Model 


Using  the  log  of  the  maximum  likelihood  function  as  given  in 


Appendix  C for  EF1,  then 

j 


and 


V ln[f  (t.  IH  )] 

ifl  ^ ^ 0 


-189.87 


2 ln[f^  (t^  !H^  )]  = -3937.13 


J 

J. 

which  yields 


(-3937.13)  - (-189.87) 


>H 


<H, 


(25) 


or 


-3747.26  ' 0 (26) 

<H 

0 

Since  -3747.26  is  less  than  0.0,  Hq  is  selected  over  H^. 
This  process  is  continued  with  the  other  models  until  one 
emerges  over  all  others. 
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However,  a decision  had  to  be  made  on  whether  or  not 
all  of  the  models  could  be  tested  against  one  another.  It 
was  quite  obvious  that  the  JM  model,  which  required  only 
time-to-error  as  input,  could  not  be  tested  against  the  LJM, 
which  required  time-to-error  and  the  number  of  errors  per 
interval  as  input.  The  only  reasonable  answer  to  be  found 
was  that  hypothesis  testing  would  be  dependent  on  input. 
Thus,  JM , SW,  JMG,and  WM1,  which  required  only  time-to-error 
as  input,  were  tested  using  the  Likelihood  Ratio  Test.  In  a 
separate  and  independent  test  the  LJM,  LSW,  MLSW,  MG  and  WM2 
models  v/ere  tested  using  the  Likelihood  Ratio  Test.  For  ease 
of  computation  the  natural  logarithm  was  used  in  computing 
the  maximum  likelihood  function  for  each  distribution.  On 
all  Program  SRMOD's  outputs  the  natural  log  of  the  maximum 
likelihood  functions  (LLR)  is  given.  In  several  cases  the 
maximum  likelihood  function  became  so  small  it  was 
considered  equal  to  zero  by  the  computer.  The  logarithm  of 
zero  cannot  be  taken  and  Program  SRMOD  aborted.  To  resolve 
this  problem  the  log  of  the  maximum  likelihood  function  was 
set  arbitrarily  to  -99999.0  whenever  the  computer  considered 
the  maximum  likelihood  function  equal  to  zero.  On  -11  of 
Program  SRMOD's  outputs  an  LLR  equal  to  -99999.0  should  be 
treated  as  an  extremely  large  negative  number  greater  than 
the  natural  logarithm  of  ( EXP (-675 . ) ) ) . A complete  listing 
of  Program  SRMOD  is  given  in  Appendix  A. 
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Chapter  VI 


Execution  And  Results 


VI. 1 Execution  of  Program  SRMOD 

Program  SRMOD  was  designed  and  developed  to  estimate 
the  parameters  for  each  of  the  software  reliability  models 
in  Table  II  and  then  discriminate  between  the  models  by 
hypothesis  testing.  For  each  of  the  time-to-error  files 
listed  in  Table  IV,  parameters  were  estimated  by  Program 
SRMOD  for  all  of  the  software  reliability  models  being 

I considered.  Next  Program  SRMOD  did  a hypothesis  test  using 

j the  Likelihood  Ratio  Test  as  described  in  Section  V.4.  Upon 

' completion,  a listing  was  printed  containing  the  vital 

I 

i statistics.  A complete  self-documented  copy  of  Program  SRMOD 

I is  contained  in  Appendix  A. 

j Data  input  requirements  for  Program  SRMOD  were 

j 

‘ fairly  simple.  Each  input  record  represented  the  time 

I between  two  successive  SPRs  and  the  number  of  errors  that 

I 

occurred  within  this  time  frame.  From  this  data  all 

calculations  for  parameter  estimation  and  hypothesis  testing 
could  be  made  for  both  Group  I and  Group  II  models. 

Calculations  for  Group  I models  ignored  the  data  field  which 
contained  the  number  of  errors  per  time  interval.  EF5C  is 
listed  in  its  entirety  in  Appendix  B as  a sample  of  the 

input  required. 
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A separate  computer  output  listing  was  created  for  ] 

each  group  for  a particular  time-to-error  file.  In  all  , 

there  were  20  Group  I output  listings  and  20  Group  II  output 
listings.  Each  listing  was  basically  of  the  same  format  and 
contained  information  such  as  the  total  number  of  errors  to 
date,  total  number  of  intervals,  estimated  parameters  for 

each  model,  the  log  of  the  maximum  likelihood  function  for 

each  function,  and  finally  the  model  selected  by  hypothesis 
testing.  Appendix  C contains  all  40  output  listings. 

A summary  of  the  results  from  Program  SRMOD  are 
presented  in  Tables  VI  and  VII.  The  following  section  will 
discuss  these  results. 

VI. 2 Results  from  Program  SRMOD  ; 

Table  VI  lists  the  time-to-error  files,  along  with 
the  software  reliability  models  that  were  selected  for  each. 

Most  obvious  in  Table  VI  is  the  number  of  files  which  had  a ^ 

] 

geometric  model  selected  for  their  time-to-error  data.  In 
Table  VII  it  can  be  seen  that  the  majority  of  files  had  a 
geometric  model  selected.  EF1  was  one  of  the  largest  time-  ■ 

to-error  files  and  was  extracted  from  Filel.  It  contained  , 

i 

data  from  152  SPRs.  The  Schick-Wolverton  Model  was  selected  | 

as  the  distribution  which  best  modelled  its  time-to-error 
data  points.  However,  when  EF2  the  operational  data  for  the 
same  software  project,  was  joined  with  EF1,  the  Jelinski- 
Moranda  Geometric  Model  was  selected.  The  data  for  both 
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"modifications”  (EF3A  and  EF3B)  of  File3  had  the  Jelinski- 
Moranda  Geometric  Model  selected.  EF4,  which  consisted  of 
149  data  points  from  File4,  had  the  Jelinski-Moranda  De- 
Eutrophication  Model  selected  for  its  distribution.  FileS 
was  divided  into  two  groups  of  seven  files.  One  group 
depicted  data  collected  by  the  day,  the  other  depicted  data 
collected  by  the  hour.  Each  of  the  seven  files  contained 
data  from  an  independent  functional  group  of  software.  The 
files  for  functional  groups  "A"  and  "D" 
(EF5A,EF5AH,EF5D,EF5DH)  were  the  only  files  which  had  a 
different  model  selected  for  data  collected  by  the  day  than 
for  data  collected  by  the  hour.  The  Jelinski-Moranda 
Geometric  model  was  selected  the  majority  of  the  time  for 
the  subfiles  of  File5.  In  total  three  different  models  were 
chosen  from  Group  I for  the  twenty  files. 

Group  II  was  more  consistent  in  its  model  selection. 
EF5GH  was  the  only  file  that  did  not  have  the  Modified 
Geometric  Model  selected  for  its  time-to-  error  data.  The 
Weibull2  model  was  selected  for  EF5GH.  The  data  within 
EF5GH  was  data  recorded  by  the  hour  for  "functional  group" 
"G"  of  File5.  No  obvious  differences  between  this  data  and 
data  in  other  files  could  be  found. 


T 


Table  VI 

Model  Selection  By  Group* 


Distribution  Selected  Distribution  Selected 


File 

From  Group  I 

From  Group  II 

EF1 

SW 

MG 

EF2 

JMG 

MG 

EF1/2 

JMG 

MG 

EF3A 

JMG 

MG 

EF3B 

JMG 

MG 

EF^l 

JM 

MG 

EF5A 

JM 

MG 

EF5B 

JMG 

MG 

EF5C 

JMG 

MG 

EF5D 

JM 

MG 

EF5E 

JMG 

MG 

EF5F 

JMG 

MG 

EF5G 

JMG 

MG 

EF5AH 

SW 

MG 

EF5BH 

JMG 

MG 

EF5CH 

JMG 

MG 

EF5DH 

JMG 

MG 

EF5EH 

JMG 

MG 

EF5FH 

JMG 

MG 

EF5GH 

JMG 

WM2 

* As  specified  in  Table  V: 

Group  I consisted  of  the  JM , SW,  WM1,  and  JMG  models 
and  required  only  the  diference  in  time 
between  "stop  dates"  as  input. 

Group  II  consisted  of  the  LJM,  LSW,  MLSW,  WM2,  and 


MG  models  and  required  time- to-error  and 
the  number  of  errors  within  each  interval 


Table  VII 


i 


1 

j 


I 


Selection  Summary 


Group  I 

Model 

Number  of 

files 

selecting 

model 

JM 

3 

sw 

2 

WM1 

0 

JMG 

15 

Group  II 


Model 

Number  of 
selecting 

files 

model 

LJM 

0 

LSW 

0 

MLSW 

0 

WM2 

1 

MG 

19 
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VI. 3 Execution  of  Program  Time-To-Fix 

The  data  for  Program  Time-To-Fix  was  used  in  three 
different  variations.  First,  each  file  in  Table  V was  used 
as  a continuous  file  to  see  if  a distribution  could  be  found 
for  the  random  variable  time-to-fix  of  an  entire  file. 
Next,  an  attempt  to  model  time-to-fix  within  the  major  error 
categories  (e.g.  "AA”,  "BB”,  etc.)  for  a given  file  was 

performed.  Finally,  time-to-fix  within  subcategories  (e.g. 
"AAOIO”,  "DDOIO”,  etc.)  was  modelled,  trying  to  determine  a 
distribution  which  best  suited  the  data  points  in  each 
subcategory  of  each  file.  There  were  20  categories  and  over 
140  subcategories  whose  time-to-fix  data  points  were 
considered  for  modelling  on  each  file  in  Table  V.  This 
means  there  were  160  output  listings  for  each  file  or  in 
total  960  listings  from  Program  Time-To-Fix.  This  was  too 
voluminous  to  include  in  this  report.  It  was  necessary  to 
condense  the  output  and  include  the  results  in  table  format, 
as  can  be  seen  in  Tables  VIII  through  XVII.  Sample  outputs 
from  Program  Time-To-Fix  are  given  in  Appendix  D. 

Each  output  listing  consisted  of  four  parts.  First 
the  estimates  for  the  parameters  were  listed  for  the  eleven 
distributions  listed  in  Table  III.  Next  the  results  from 
the  Kolmogorov-Smirnof f test  were  printed.  The  Likelihood 
Ratio  Test  results  followed  and  then  finally  the  decision  of 
which  model,  if  any,  was  printed. 

Input  for  Program  Time-To-Fix  consisted  only  of  the 
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time-to-fix  and  the  error  category  for  that  particular 
problem.  As  an  example  of  the  input  required  for  Program 
Time-To-Fix  a portion  of  FF1  is  listed  in  Appendix  E. 

VI.  4 Results  from  Program  Time-To-Flx 

Time-to-fix,  without  regard  to  an  error  category, 
could  not  be  modelled  at  the  0.20  signifinance  level  for  the 
given  files.  The  Kolmogorov-Smirnof f test  rejected  all 
distributions  when  the  files  were  used  as  a whole.  No  other 
signifinance  levels  were  tried,  primarily  because  of  the 
time  constraints  of  this  study. 

However,  when  the  major  error  categories  were 
extracted  from  the  files  and  the  time-to-fix  data  points  for 
each  category  were  modelled,  distributions  were  selected. 
For  example,  there  were  109  category  "AA”  errors  in  file 
FF5B  and  the  Program  Time-To-Fix  selected  the  log-normal 
distribution  as  the  distribution  that  best  modelled  time-to- 
fix  for  this  error  category.  However,  there  were  several 
categories  for  which  time-to-fix  could  not  be  modelled. 
There  were  192  category  "AA”  errors  within  file  FF1,  but  no 
distribution  was  selected. 

Table  VIII  reflects  the  percentage  of  categories 
which  had  distributions  selected  for  their  time-to-fix  data. 
As  can  be  seen,  there  was  a wide  difference  between  files. 
Categories  "HH",  "II",  "KK",  "PP",  "RR",  and  "SS"  had 
distributions  selected  for  all  six  files.  Time-to-fix  for 


some  categories  could  not  be  modelled  because  there  was 
either  zero  or  only  one  data  point.  Time-to-fix  for  any 
particular  category  was  always  modelled  in  at  least  one 
file,  but  never  was  the  distribution  the  same  for  all  six 
files.  The  number  of  data  points  per  category  for  each  file 
varied  from  zero  to  572.  Table  IX  shows  that  the  number  of 
data  points  per  category  also  varied  greatly.  A large 
percentage  of  categories  had  more  than  twenty  data  points, 
which  means  that  the  findings  could  be  statistically 
significant.  As  would  be  expected,  those  categories  with 
very  few  data  points  passed  the  Kolmogorov-Smirnof f Test 
more  often  than  not.  However,  much  more  confidence  could  be 
placed  in  selected  distributions  which  had  a large  number  of 
data  points  and  passed  the  Kolmogorov-Smirnof f test,  than 


for  those  which  only  had  a few  data  points.  Table  X shows 
that  even  when  the  categories  had  over  twenty  time-to-fix 
data  points,  time-to-fix  could  be  modelled.  Even  the  highest 
range  of  data  points  (50  or  over)  had  distributions  selected 
for  time-to-fix  for  45  percent  of  the  categories. 

A few  examples  of  distributions  selected  are  given 
in  Table  XI.  Certain  distributions  appearred  to  be  more 
prevalent.  The  log-normal  distribution  was  chosen  42 
percent  of  the  time,  as  indicated  in  Table  XII.  The  log- 
normal, the  exponential,  and  the  gamma  distributions  were 
usually  selected  on  the  basis  of  a large  number  of  data 
points.  On  the  other  hand,  the  uniform  and  the  pareto 
distributions  were  usually  selected  on  the  basis  of  less 
than  ten  data  points. 
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As  can  be  seen  by  the  number  of  subcategories  in 


Appendix  F,  categories  such  as  "AA",  "BB”,  "CC”,  and  "DD” 
are  quite  inclusive  in  the  type  of  errors  that  they  cover. 
Trying  to  model  time-to-fix  within  categories  which  are  so 
broad  in  nature  may  not  be  the  best  approach.  Possibly 
using  subcategories,  instead  of  major  error  categories, 
could  produce  more  revealing  information  on  the  attributes 
of  time-to-fix. 

The  problem  with  this  approach  is  the  loss  of  a 
large  portion  of  the  data,  as  is  pointed  out  in 
Tables  XIII  and  XIV.  There  was  a significant  loss  of  data 
points  in  all  six  files.  Time-to-fix  could  not  be  modelled 


for  fifty  percent  of  the  864  possible  subcategory  files, 
because  the  subcategories  had  one  or  less  time-to-fix  data 
points  and,  therefore,  distributions  could  not  be  selected. 
When  there  were  enough  data  points  for  modelling,  the 
percentage  of  subcategories  having  distributions  selected 
decreased  drastically  as  the  number  of  points  increased  (See 
Table  XV).  Again,  a variety  of  distributions  was  selected. 
There  were  a significant  number  of  cases  where  distributions 
were  selected  for  all  six  files  for  a particular 
subcategory.  However,  in  no  case  was  this  distribution  the 
same  for  all  six  files.  A sample  of  the  distributions 
selected  for  the  subcategories  is  given  in  Table  XVI.  The 
entries  in  Table  XVII  were  selected  on  the  basis  of  their 
large  number  of  data  points.  Again,  the  log-normal,  the 
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exponential,  and  the  gamma  distributions  were  generally 
accepted  on  the  basis  of  a large  number  of  data  points. 

Table  XVII  points  out  that  the  uniform  and  pareto 
distributions  were  selected  more  than  any  of  the  other 
distributions.  A closer  look  into  this  fact  showed  that  the 
uniform  was  usually  selected  on  the  basis  of  fewer  than  ten 
data  points.  The  pareto  was  selected  approximately  half  the 
time  on  the  basis  of  ten  or  fewer  data  points  and  the  other 
half  of  the  time  on  the  basis  of  more  than  ten  time-to-fix 
data  points. 

No  significant  difference  was  seen  between  those 
files  measured  in  work  days  versus  calendar  days  or  calendar 
days  versus  hundredths  of  hours  to  fix.  In  a few  cases  the 
distributions  differed,  but  generally  the  distributions 
stayed  the  same. 


Table  VIII 


Modelling  Time-To-Fix  Within  Categories 


File 

Percentage  of  categories 
in  which  a distribution 
was  selected  for  time-to-fix 

FF1 

47 

FF3 

82 

FF4A 

53 

FF^B 

47 

FF5 

82 

FF5H 

88 

Overall 

67 

65 


Table  IX 


Data  Point  Spread 

Within  Categories 

Number  of  data  points 

Percentage  of  categories 

0-1 

6 

2-9 

20 

10-19 

27 

20-49 

21 

50-over 

26 

Table  X 


Modelling  of  Time-To-Fix  Within  Categories  by  the  Number  of 

Data  Points 


Percentage  of  categories  ;] 

in  which  a distribution  was  l\ 

Number  of  data  points  selected  for  time-to-fix 


Table  XI 


A Sample  of  Distributions  Selected  For  Time-To-Fix  Within 

Categories 


Distribution 


File 

Category 

Data  points 

selected 

FI 

JJ 

152 

Gamma 

F3 

BB 

51 

Lognormal 

F4A 

CC 

18 

Gamma 

F4B 

KK 

31 

Lognormal 

F5 

AA 

109 

Lognormal 

F5 

PP 

148 

Gamma 

F5 

RR 

144 

Exponential 

F5H 

MM 

67 

Lognormal 

Table  XII 


Distribution  Selection  For  Time-To-Fix 

Categories  ; 

Within  Major  Error 

Distribution 

Percentage  of 
time  selected 

Lognormal 

42 

Exponential 

18 

Un i form 

15 

Pareto 

1 1 

Gamma 

9 

Extreme-V£.lue  Largest 

3 

Extreme-Value  Smallest 

1 

Weibull 

1 

67 


Table 


XIII 


Modelling  Time-To-”ix  Within  Subcategories 


File 

Percentage  of  subcategories  in 
which  a distribution  was  selected 
for  time-to-fix 

FFl 

67 

FF3 

30 

FF4A 

34 

FF4B 

33 

FF5 

50 

FF5H 

46 

Overall 

43 

Table  XIV 

Data  Point  Spread 

Within  Subcategories 

Number  of  data  points 

Percentage  of  subcategories 

0-1 

50 

2-9 

29 

10-19 

10 

20-49 

7 

50-over 

4 

68 


Table  XV 


Modelling  Titne-To-Fix  Within  Subcategories  by  the  Number  of 

Data  Points 


Percentage  of  subcategories 
in  which  a distribution  was 

Number  of  data 

points 

selected  for  time-to-fix 

0-1 

0 

2-99 

96 

10-19 

89 

20~i\9 

79 

50  or 

over 

27 

Table  XVI 


A Sample 

of  Distributions 
Error 

Selected  For 
Subcategories 

Time-To-Fix  Within 

Distribution 

File 

Subcategory 

Data  points 

selected 

FI 

BB100 

54 

Log-normal 

FI 

CC050 

60 

Log-normal 

FI 

GG020 

36 

Log-normal 

FitA 

BB020 

46 

Gamma 

F^A 

DD140 

33 

Gamma 

F4B 

DD010 

27 

Log-normal 

F5 

AA040 

40 

Log-normal 

F5 

BB060 

173 

Gamma 

F5 

DD050 

67 

Exponential 

F5 

PP020 

1 16 

Exponential 

F5H 

AA040 

40 

Exponential 

F5H 

BB020 

46 

Gamma 

F5H 

PP020 

1 16 

Log-normal 

59 


rr 


Table  XVII 

Distribution  Selection  For  Time-To-Fix  Within  Error 

Subcategories 


t! 


Percentage  of 
Distribution  time  selected 


Uniform  - 36 
Pareto  20 
Log-normal  19 
Exponential  13 
Gamma  5 
Extreme-Value-Largest  3 
Extreme-Value  Smallest  1 
Laplace  1 
Weibull  1 
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Chapter  VII 


Conclusions  and  Recommendations  * 


VII. 1 Conclusions 

Before  beginning  this  thesis  it  was  not  known  if  the 
existing  software  reliability  models  could  be  tested  against 
one  another.  It  was  found  that  certain  models,  as  given  in 
Table  II,  could  be  tested  and  the  criteria  for  which  models 
could  or  could  not  be  tested  was  primarily  based  on  the 
available  data  and  the  basic  underlying  principles  of  each 
model.  Utilizing  maximum  likelihood  estimators  for 
parameter  estimations  and  the  Likelihood  Ratio  Test  for 
hypothesis  testing,  a computer  program  was  written  which 
could  discriminate  between  the  software  reliability  models 
for  a given  data  set.  One  of  the  most  important  findings 
was  the  dire  need  for  correct  and  usable  input  data  before 
testing  can  begin. 

The  Jelinski-Moranda  Geometric  Model  was  chosen  a 
greater  number  of  times  than  any  other  model  in  Group  I of 
Table  II.  The  Modified  Geometric  Model  showed  even  more  of 
a predominance  over  Group  II  models.  Why  did  this  occur? 

Both  geometric  models  allow  an  infinite  number  of  errors  to 
occur  over  the  life-cycle  of  a computer  software  system. 

Some  of  the  other  models  assume  a fixed  number  of  errors.  j 

I 

I 

Allowing  for  an  infinite  number  of  errors  is  a more 

J 
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realistic  view  of  the  data  files  under  study.  The  data 
files  came  from  projects  which  were  large  and  had  numerous 
changes  and  updates.  New  errors  were  introduced  throughout 
the  life-cycle  and  new  requirements  placed  the  projects  in  a 
dynamic  environment.  This  was  noticed  when  File2  was  joined 
with  Filel.  The  addition  of  the  operational  data  from  File2 
onto  the  testing  data  from  Filel  altered  the  selection  of 
models  from  the  Schick-Wolverton  Model  for  Filel  to  the 
Jelinski-Moranda  Geometric  Model  for  File1/2.  The  detection 
rate  of  the  geometric  models  varies  geometrically  and  is  not 
fixed  as  in  some  of  the  other  models.  This  feature  could 
account  for  the  variation  in  the  severity  of  testing  at  any 
particular  time  and  also  compensating  for  errors  that  do  not 
necessarily  have  an  equal  probability  of  being  detected. 
The  size  of  the  files  had  no  impact  on  whether  or  not  the 
geometric  models  were  selected.  This  would  enable  them  to 
be  applied  to  any  size  software  project. 

Every  data  base  should  be  be  given  a chance  to  be 
modelled  and  not  to  be  prejudged  regarding  its  distribution. 
Saying  that  all  t ime- to-error  data  should  be  modelled  by  the 
geometric  models  could  not  be  stochastically  supported.  A 
computer  program,  such  as  Program  SRMOD,  may  be  required  to 
discriminate  between  models  as  a software  project  proceeds. 
The  addition  of  EF2  onto  EF1  to  form  EF1/2  is  a good  example 
of  this.  One  model  was  chosen  for  the  testing  phase,  while 
another  model  was  chosen  for  the  combined  data  from  the 
testing  and  operational  phases.  It  may  be  necessary  to  use 


several  different  models  during  the  life  of  the  software 
package . 

Modelling  time-to-fix  was  not  quite  as  conclusive  as 
modelling  time-to-error . It  appeared  that  distributions  for 
time-to-fix  were  project  dependent.  This  could  very  well 
have  been  caused  by  the  error  categorization  methodology.  A 
certain  error  category  could  mean  one  thing  to  one 
contractor,  but  quite  another  to  another  contractor.  Herein 
lies  the  problem  of  error  categorization. 

Disregarding  error  categories  and  trying  to  model 
software  time-to-fix  at  large  was  unsuccessful  at  the  0.20 
significance  level.  Modelling  time-to-fix  within  major 
categories  looked  quite  promising,  in  that  time-to-fix  for 
numerous  categories  had  distributions  selected.  Trying  to 
model  time-to-fix  within  subcategories,  did  not  improve  the 
results.  Rather,  many  data  points  were  lost  and  could  not 
be  used  in  testing. 

For  many  data  sets  the  log-normal  distribution  was 
chosen.  It  is  quite  probable  that  with  further  modelling 
studies  the  log-normal  distribution  may  play  a key  role  in 
modelling  time-to-fix  for  software  projects.  Other 
distributions  that  were  selected  quite  frequently  are  the 
exponential,  gamma,  pareto,  and  uniform.  Again,  further 
testing  and  software  error  collection  endeavors  must  b^ 
performed  to  validate  or  reject  any  of  the  findings  of  this 
study. 

At  best  with  the  data  that  is  presently  available  it 


appears  that  modelling  time-to-fix  is  software  project 
dependent.  No  distribution  emerged  as  the  best  for  any 
particular  file,  category,  or  subcategory.  Possibly  with 
the  use  of  the  newer  error  categories  better  results  would 
have  been  obtained. 

V I I . 2 Recommend ations 

A disease  can  not  be  cured  without  the  knowledge  of 
the  cause.  So  is  the  case  with  unreliable  software.  Herein 
lies  the  importance  of  collecting  software  error  data.  How 
can  the  dilemma  of  unreliable  software  ever  be  solved,  if 
its  characteristics  are  not  identified  and  understood?  More 
and  better  error  collection  tools  and  techniques  must  be 
developed  and  used.  Errors  should  be  collected  on  all  types 
of  software  projects  involving  higher  order  languages, 
assembly  languages,  batch  mode,  time-sharing,  structured 
design,  and  the  countless  other  software  procedures. 
Collection  of  software  error  data  may  seem  costly,  but 
considering  that  from  software  error  data  such  items  as 
personnel  training  requirements,  pinpointing  problem 
programs,  time-to-fix  analysis,  and  software  reliability,  to 
mention  a few,  may  ultimately  bo  determined,  by  its 
collection  and  use. 

The  ultimate  test  for  software  reliability  models  is 
their  actual  use  during  software  develooment  and 
implementation.  Only  then  can  their  validity  be  proven.  If 
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software  reliability  models  are  used  as  a project  is  being 
developed,  there  is  more  control  over  what  data  should  be 
collected  and  the  data's  correctness.  Again,  the  importance 
of  continued  error  collection  is  stressed.  Further  analysis 
is  required  to  validate  the  findings  of  this  study.  More 
and  better  data  must  be  collected,  so  that  other  software 
reliability  models  can  be  included  in  a program  such  as 
Program  SRMOD.  A software  manager  could  then  use  this 
program,  or  one  similar  to  it,  to  select  a software 
reliability  model  for  a software  package.  From  the  model 
selected  the  software  manager  could  determine  the 
reliability  of  the  software  and  its  mean-time-to-error . On 
the  basis  of  these  calculations,  decisions,  such  as  whether 
or  not  to  accept  the  software,  whether  or  not  to  implement 
the  software,  or  whether  or  not  the  software  package  will  be 
completed  on  schedule,  could  be  made.  Of  the  two  groups  of 
software  reliability  models  listed  in  Table  II,  how  does  the 
software  manager  know  which  group  of  models  to  use?  One 
suggestion  is  that  if  the  manager  is  concerned  only  in  the 
occurrence  of  the  software  downtime,  regardless  of  how  many 
errors  occurred  in  the  interval,  the  model  selected  from 
Group  I should  be  used,  otherwise  the  model  from  Group  11 
should  be  used. 

In  this  study  entire  projects  or  subfunctions  were 
examined  for  their  reliability.  A possible  different 
approach  would  be  to  determine  the  reliability  of  each 
computer  program  and  utilizing  the  software  system 
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flowcharts  a topological  network  of  programs  could  be 
formed.  Each  program  could  use  a software  reliability  model 
for  determining  its  reliability,  but  the  entire  network's 
reliability  would  be  based  upon  the  principles  of 
reliability  decomposition.  Another  recommendation  is  to 
investigate  and  include  into  Program  SRMOD,  if  possible,  the 
Geometric  Poisson  Model  referenced  in  Section  III. 1.4. 

Modelling  time-to-fix  appears  to  be  dependent  on  the 
standarization  of  error  categorization.  The  exact 
definition  of  error  categories  must  be  standardized  before 
any  meaningful  results  can  be  obtained.  From  the  findings 
of  this  study  it  appears  very  unlikely  that  a distribution 
will  ever  be  found  which  will  universally  model  software 
time-to-fix.  It  does  seem  feasible  to  model  time-to-fix  data 
within  error  categories  for  a specified  software  project. 
As  mentioned  earlier,  this  study  used  a significance  level 
of  0.20.  An  investigation  at  perhaps  a significant  level  of 
0.10  or  0.05  may  be  worthwhile.  Another  interesting  area 
that  should  be  examined  is  the  correlation  between  the 
criticality  of  an  error  and  the  amount  of  time  it  took  to 
fix  the  error.  Both  items  are  available  in  the  RADC  files. 

If  one  recommendation  had  to  be  chosen  over  all 
others,  it  would  be  the  importance  of  the  continuation  of 
software  error  collection.  A list  of  items  that  would  have 
aided  this  investigation  and  that  should  be  considered  in 
future  software  collection  endeavors  is  as  follows: 
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1)  Instruction  mix 


} 

i 

1 

* 
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2)  Processing  rate 

3)  CPU  time  for  each  test  of  a computer  program 
ij)  Number  of  machine  instructions 

5)  Number  of  source  instructions 

6)  Number  of  tests  for  each  computer  program 

7)  The  number  of  successful  and  unsuccessful  tests 
and  the  order  in  which  they  occurred 

8)  Apriori  probability  of  correcting  a specific  error 

9)  Date  error  occurred 

10)  Number  of  errors  occuring  during  each  test 

11)  Date  error  was  fixed 

12)  Controls  allowed  over  program  inputs 

13)  l^as  seeding  and  tagging  performed?  If  so,  how  many 
indigenous  and  induced  errors  were  found  per  test 

14)  Exact  dates  of  when  the  development,  design, 
coding,  testing,  integration,  and  operational 
phases  began 

15)  Error  categories  with  definitions  that  are 
unambiguous 

16)  Phase  in  which  error  occurred 

17)  Program  in  which  error  occurred 

18)  Criticality  of  error 

19)  The  length  of  time  in  actual  CPU  seconds 

and  the  number  of  days  it  took  to  fix  an  error 

20)  Intensity  of  testing 

21)  Manpower  allotted  for  the  software  project 

22)  Experience  of  manpower  allotted 

Of  course,  it  is  realized  that  many  of  these  items 
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are  quite  subjective  in  nature,  and  the  cost  of  collecting 
all  of  these  characteristics  could  be  prohibitive.  The 
economics  and  feasibility  must  be  weighed  for  any  given 
software  development  project.  Only  with  further  error  data 
can  there  be  any  hope  of  quantifying  the  problems  associated 
with  the  development  and  operation  of  software. 


Appendix  A 
Program  SRMOD 


Contained  within  this  appendix  is  Program  SRMOD, 
which  was  used  for  parameter  estimation  and  testing  of 
software  reliability  models.  The  program  is  self-documented 
and  was  written  in  FORTRAN  for  execution  on  a CDC6600.  ANSI 
standards  were  adhered  to  as  much  as  possible  to  aid  in 
portability.  The  read  and  write  statements  may  require 
modifications  before  the  program  can  be  used  on  another 
computer  system.  Also,  subroutine  ZBRENT  from  the  IMSL 
subroutine  library  was  used  to  compute  the  root  of  "k”  for 
the  key  equation  in  subroutines  JMG  and  MG.  If  the  IMSL 
subroutine  library  is  not  available  on  the  executing 
computer  system  a subroutine  that  can  compute  the  root  of  a 
polynomial  will  have  to  be  obtained  and  substituted  in  for 
ZBRENT.  The  memory  requirement  of  56K  is  for  a large  data 
base  consisting  of  2200  error  records.  If  the  data  base  to 
be  tested  is  not  that  large,  many  of  the  matrices  utilized 
in  the  program  can  be  reduced,  thereby  reducing  memory 
requirements.  Central  processing  (CP)  time  varies 
considerably  and  is  not  dependent  on  the  number  of  input 
records,  but  rather  on  the  convergence  rate  of  the  key 
equations  used  in  parameter  estimations.  For  the  data  bases 
utilized  in  this  study  the  CP  time  varied  from  less  than  1 
second  to  50  seconds. 
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Given  below  is  the  jobstreara  required  to  execute 
Program  SRMOD  from  source  code  on  a CDC6600  computer.  It  is 
assumed  that  the  software  error  input  file  is  stored  on  disk 


as  Tapel . 

CAS,CM56000.  T770293 , CASTLE , BOX  4534 
ATTACH, IMSL, I D=X654321 
LIBARY, IMSL. 

ATTACH, TAPE1 . 

FTN. 

LGO. 

7/8/9  CARD 

SOURCE  DECK  OF  PROGRAM  SRMOD 
7/8/9  CARD 

6/7/8/9  CARD  (END  OF  JOB) 
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CALL  HYP0(LR>5»9» J9)  001820 
CALL  PRINT<PARAM»HEADER»5»9» J9»FLAG»TNE»LR)  ' 001830 
STOP  001840 
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Appendix  B 

Sample  Input  for  Program  SRMOD 

V 

Given  within  this  appendix  is  file  F5C  in  its 
entirety  and  it  represents  typical  input  for  Program  SRMOD. 
The  first  entry  of  each  record  indicates  the  time  interval 
between  successive  software  problem  reports.  The  second 
entry  represents  the  number  of  errors  occurring  within  this 
time  interval. 


1 10 


f 


Appendix  C 

Program  SRMOD  Output 

Within  this  appendix  is  the  output  from  Program 
SRMOD  for  each  of  the  twenty  input  files.  Each  output  is 
labelled  as  to  which  error  file  it  pertains  to.  There  are 
two  outputs  for  each  file.  One  for  Group  I models  and  one 
for  Group  II  models.  "LLR”  represents  the  natural  logrithra 
of  the  maximum  likelihood  function.  In  some  cases  the 
maximum  likelihood  function  became  so  small  it  was 
considered  equal  to  zero  by  the  computer  and  subsequently 
the  logarithm  could  not  be  taken.  To  resolve  this  situation 
”LLR”  was  set  equal  to  the  arbitrary  number  -999999.0 
whenever  the  logarithm  could  not  be  taken  for  this  reason. 
The  next  40  pages  are  the  actual  outputs  from  Progam  SRMOD. 
All  parameters  are  the  same  as  those  used  in  the  text  of 
this  thesis. 
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\ ^ ' file:  EFl 

i EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 

‘ REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 2172. 

TOTAL  NUMBER  OF  INTERVALS  = 152. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 


JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  248.0010000  PHI'=  .004852014  LLR=  -184.28 


SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  152.0010000  PHI=  .018812043  LLR=  -127.13 


JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  1.1727845  K=  .994996064  LLR=  -185.34 


UEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  3.5433276  BETA=  1.288208006  LLR=  -1000227.17 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED.* 
SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 


file:  EFl 


EVALUATION  OF  GROUP  II  SOFTUARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 2172. 

TOTAL  NUMBER  OF  INTERVALS  = 152. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE! 

LIPOW-JELINSKI-MORANDA  SOFTUARE  RELIABILITY  MODEL 
N=  2546.1000000  PHI  = .009734999  LLR=  -1811.59 

LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  2172.1000000  PHI=  .024776264  LLR=  -3937.13 

MODIFIED-LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  2176.1000000  PHI=  .000306270  LLR=  -1816.24 

MODIFIED  GEOMETRIC  SOFTUARE  RELIABILITY  MODEL 
D=  .7792413  K=  1.000000225  LLR=  -189.87 

WEIBULL  SOFTUARE  RELIABILITY  MODEL 

ALPHA=  8.1579818  BETA=  1.104085712  LLR=  -99999.00 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTUARE  RELIABILITY  MODEL 
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file:  EF2 

EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 198. 

TOTAL  NUMBER  f INTERVALS  = 86. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  AREt 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000fT0TAL  NUMBER  OF  tSTOPS*)** 

N=  10086.0010000  PHI=  .000034624  LLR=  -176.84 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  ♦STOPSf)** 

N=  10086.0010000  PHI=  .000005879  LLR=  -278.70 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .1843821  K=  1.017790064  LLR=  -166.95 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.6343261  BETA=  2.564747416  LLR-  -274.50 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 


file:  EF2 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 198. 

TOTAL  NUMBER  OF  INTERVALS  = 86. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  tSTOPS*)** 

N=  10198.1000000  PHI  = .000079119  LLR=  -217.30 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000TT0TAL  NUMBER  OF  tSTOPS*)** 

N=  10198.1000000  PHI=  .000013402  LLR=  -517.36 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  997.1000000  PHI=  .000007243  LLR=  -212.46 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .3481729  K=  1.000000225  LLR=  -176.73 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA*  1.7505832  BETA*  2.317399574  LLR=  -633.13 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 

« « 
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file:  EFl/2 


EVALUATION  OF  GROUP  I SOFTWARE  RELIADILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 2370. 

TOTAL  NUMBER  OF  INTERVALS  = 238. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  362.0010000  PHI=  .002594701  LLR=  -357.52 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  263.0010000  PHI=  .001593007  LLR=  -1000157.19 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  1.1039887  K=  .994998539  LLR=  -355.86 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  2.2821522  BETA=  1.658591296  LLR=  -1000139.45 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 


file:  EFl/2 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  LF  ERRORS  TO  DATE  = 2370. 

TOTAL  NUMBER  OF  INTERVALS  = 233. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE.* 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  2401.1000000  PHI  = .010385982  LLR=  -2012.38 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  2370.1000000  PHI=  .006958987  LLR=  -1000229.81 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  2370.1000000  PHI=  .000131312  LLR=  -1979.38 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .5774299  K=  1.000000225  LLR=  -368.62 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  4.5476435  BETA=  1.213738644  LLR=  -104017.74 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
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file:  EF3A 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 144. 

TOTAL  NUMBER  OF  INTERVALS  = 54. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 


JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

**CONVERGENCE  WAS  NOT  REACHED*:)? 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  1 STOPS* )♦♦ 

N==  10054.0010000  PHI=  .000033415  LLR=  -113.04 


SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  1 STOPS*)** 

N®  10054.0010000  PHI=  .000004598  LLR=  -190.19 


JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .1882861  K=  1.027401418  LLR=  -105.43 


WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.5693922  BETA=  2.515452292  LLR=  -183.00 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF3A 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 

TOTAL  NUMBER  OF  INTERVALS  = 54. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 


LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED*^ 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  tSTOPS#)** 

N=  10144.1000000  PHI  = .000088541  LLR=  -150.17 


LIPOW-SCHICK-UOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ♦STOPS* )*♦ 

N=  10144.1000000  PHI=  .000012158  LLR=  -371.43 


MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

CN=10000+T0TAL  NUMBER  OF  ♦STOPS* )♦♦ 


N=  10144.1000000  PHI=  .000001103  LLR=  -136.65 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .3354005  K=  1.000000225  LLR=  -112.99 

UEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.8409500  BETA=  1.995204713  LLR=  -598.27 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
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file:  EF3B 


EVALUATION  OF  GROUP  I S50FTUARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 75. 

TOTAL  NUMBER  OF  INTERVALS  = 30. 

I 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE; 


JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦ ♦CONVERGENCE  WAS  NOT  REACHED:!'* 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ♦STOPS#)** 

N=  10030.0010000  PHI=  .000034818  LLR=  -61. AO 


SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ♦STOPS#)** 

N=  10030.0010000  PHI=  .000009208  LLR=  -82.15 


JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .2170610  K=  1.037888981  LLR=  -59.65 


WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.5983478  BETA=  2.735939264  LLR=  -74.64 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED; 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 


file:  EF3B 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 75. 

TOTAL  NUMBER  OF  INTERVALS  = 30. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARET 


LIPOU-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED*:*; 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  ESTOPS*)** 

N=  10075.1000000  PHI  = .000006001  LLR=  -79.13 


LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  #STOPS*)** 

N=  10075.1000000  PHI=  .000022932  LLR=  -174.71 


MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  115.1000000  PHI=  .000277906  LLR=  -78.93 


MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .3488373  K=  .999999994  LLR=  -61.59 


UEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.3858821  BETA--=  3.162500962  LLR=  -200.62 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
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file:  EF4 

EVALUATION  OF  GROUP  I SOFTUARE  RFLIAPILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

i TOTAL  NUMDER  OF  ERRORS  TO  DATE  = 1254. 

I TOTAL  NUMBER  OF  INTERVALS  = 435. 

i PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  AREt 

' JELINSKI--MORANDA  SOFTWARE  RELIABILITY  MODEL 

I 'N=  604.0010000  PHI=  .001086245  LLR=  -837.44 

i 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  436.0010000  PHI=  .001028449  LLR-  -1000688,93 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .6017035  K==  .997989480  LLR=  -845.95 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

' ALPHA=  1.6803426  BETA=  2.538534110  LLR=  -1246.52 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
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file:  EF4 

EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 1254. 

TOTAL  NUMBER  OF  INTERVALS  = 435. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  AREt 

LIPOU-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  1571.1000000  PHI  = .001362447  LLR^  -1204.93 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  1256.1000000  PHI=  .001109199  LLR=  -2600.13 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=»  1259.1000000  PHI=  .000007872  LLR=  -1179.66 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .3714118  h=  1.000000225  LLR=  -865.78 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.8039613  BETA=  2.408449833  LLR=  -3503.76 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED.’ 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 

♦ 

\ 
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file:  EF5A 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 432. 

TOTAL  NUMBER  OF  INTERVALS  = 149, 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE.* 

JELINSKI-^ 3RANDA  SOFTWARE  RELIABILITY  MODEL 

N=  931.0010000  PHI=  .000476236  LLR=  -232.72 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  694.0010000  PHI=  .000300735  LLR=-  -303.79 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .4395602  K=  .998973879  LLR=  -282.74 

UEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.9040927  BETA=  2.576061061  LLR=  -310.03 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
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file:  EF5A 

EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 432. 

TOTAL  NUMBER  OF  INTERVALS  ==  149. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOU-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  1080.1000000  PHI  - .001391441  LLR=  -408.04 

LIPOW-SCHICK-WOLUERTON  SOFTWARE  RELIABILITY  MODEL 
N=  987.1000000  PHI=  .000714475  LLR=  -861.16 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  454.1000000  PHI=  .000044175  LLR=  -420.13 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .4070826  K=  1.000000225  LLR=  -282.90 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.9463397  BETA=  2.729025073  LLR=  -832.35 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 


file:  EF5B 

EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 293. 

TOTAL  NUMBER  OF  INTERVALS  = 1A6. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE! 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦ ♦CONVERGENCE  WAS  NOT  REACHEDSc* 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ♦STOPS*)** 

N=  10293.1000000  PHI  = .000038707  LLR=  -429.08 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  tSTOPS*)** 

N=  10293.1000000  PHI=  .000003599  LLR=  -920.82 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  915.1000000  PHI=  .000001399  LLR=  -436.78 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .1967605  K=  1.000000225  LLR=  -383.36 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.4295102  BETA=  3.463712479  LLR=  -832.92 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED.* 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 


♦ • 


file:  EF5C 


EVALUATION  OF  GROUP  I SOFTUARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 80. 

TOTAL  NUMBER  OF  INTERVALS  - 51. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTUARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  UA3  NOT  REACHED** 

♦♦CALCULATIONS  UERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  IhSTOPSi)** 

N=  10051.0010000  PHI=  .000013889  LLR=  -151.54 

SCHICK-WOLVERTON  SOFTUARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  UAS  NOT  REACHED** 

♦♦CALCULATIONS  UERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  tSTOPSt)** 

N=  10051.0010000  PHI=  .000001064  LLR=  -213.77 

JELINSKI-MORANDA  GEOMETRIC  SOFUARE  RELIABILITY  MODEL 
D=  .0756303  K=  1.028539567  LLR=  -146.74 

WEIBULL  SOFTUARE  RELIABILITY  MODEL 

ALPHA*  1.2148178  BETA*  6.199801215  LLR=  -161.05 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  GEOMETRIC  SOFUARE  RELIABILITY  MODEL 
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♦ ♦ 


file:  EF5C 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIAEa’LITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 80. 

TOTAL  NUMBER  OF  INTERVALS  = 51. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED^* 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000ETOTAL  NUMBER  OF  tSTOPS*)** 

N=  10080.1000000  PHI  = .000021749  LLR-  -157.61 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  *STOPS*)** 

N=  10080.1000000  PHI=  .000001664  LLR-  -299.87 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  140.1000000  PHI=  .000012506  LLR=  -157.50 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .1393443  K=  .999999993  LLR=  -151.51 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.2678031  BETA=  5.535613459  LLR=  -243.87 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
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file:  EF5D 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 170, 

TOTAL  NUMBER  OF  INTERVALS  = 83. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  112.0010000  PHI=  .004527282  LLR=  -182.51 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  89.0010000  PHI==  .002273557  LLR=  -203.23 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .5583315  K=  .985014918  LLR=  -182.75 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.5293342  BETA=  3.509133908  LLR=  -204.49 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
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file:  EF5II 


EVALUATION  OF  GROUP  II  SOFTUARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 170. 

TOTAL  NUMBER  OF  INTERVALS  = 83. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTUARE  RELIABILITY  MODEL 
N=  206.1000000  PHI  = .005781998  LLR=  -210.52 

LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  175.1000000  PHI=  .002899614  LLR=  -403.16 

MODIFIED-LIPOW-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  171.1000000  PHI=  .000108849  LLR=  -226.90 

MODIFIED  GEOMETRIC  SOFTUARE  RELIABILITY  MODEL 
D=  .2803987  K=  1.000000225  LLR=  -188.54 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.5812815  BETA=  3.512632233  LLR=  -394.68 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTUARE  RELIABILITY  MODEL 
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file:  EF5E 


EVALUATION  OF  GROUP  I SOFTWARE  RELIAEILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 223. 

TOTAL  NUMBER  OF  INTERVALS  = 109. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 


♦♦CONVERGENCE  WAS  NOT  REACHED** 
♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ♦STOPS* )♦♦ 


N= 


10109.0010000  PHI=  .000024622  LLR= 


-261.17 


SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 


♦♦CONVERGENCE  WAS  NOT  REACHED** 
♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  *STOPS*)** 


N= 


10109.0010000  PHI=  .000002944  LLR= 


-401.67 


JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .1935082  K=  1.004856851  LLR=  -259.51 


WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.3986271  BETA=  3.482394209  LLR= 


-317.98 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED! 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF5E 


EVALUATION  OF  GROUP  II  SOFTUARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 223. 

TOTAL  NUMBER  OF  INTERVALS  = 109. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 


LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED^* 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  iSTOPS#)^* 

N=  10223.1000000  PHI  = .000050097  LLR=  -297.24 


LIPOW-SCHICK-WOLVERTON  SOFTUARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED*^ 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ♦STOPS*)^^ 

N=  10223.1000000  PHI=  .000005970  LLR=  -661.09 


MODIFIED-LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  248.1000000  PHI=  .000023390  LLR=  -296.07 


MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .2477213  K=  1.000000225  LLR=  -261.10 


WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA*  1.4959293  BETA*  3.489720447  LLR*  -641.54 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
♦EOR 
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file:  EF5F 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 207. 

TOTAL  NUMBER  OF  INTERVALS  = 102. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 


JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

♦^CONVERGENCE  WAS  NOT  REACHED*^ 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  iSTOPSI)** 

N=  10102.0010000  PHI=  .000021590  LLR=  -257.84 


SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  tSTOPS*)** 

N=  10102.0010000  PHI=  .000003736  LLR=  -327.59 


JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .2116517  K=  1.000498749  LLR=  -257.82 


UEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.3914725  BETA=  4.415219075  LLR=  -274.46 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF5F 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 207. 

TOTAL  NUMBER  OF  INTERVALS  = 102. 

PARAMETER  ESTIMATIONS  FDR  THE  TESTED  MODELS  ARE: 

LIPOU-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  425.1000000  PHI  = .001412641  LLR=  -293.35 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  925.1000000  PHI=  .000093097  LLR^  -568.81 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  214.1000000  PHI=  .000030102  LLR=  -298.58 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .2170157  K=  1.000000225  LLR=  -257.83 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHAS  1.5045255  BETA=  4.179165058  LLR=  -525.59 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED.* 

♦ 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 


) 

1 


file:  EF5G 


EVALUATION  OF  GROUP  I SOFTWARE 
REQUIRING  ONLY  TIME  TO  ERROR 


RELIAIHLITY  MODELS 
AS  INPUT  DATA 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 1984 

TOTAL  NUMBER  OF  INTERVALS  ■=  80. 


PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 


JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

*»CONVERGENC£  WAS  NOT  REACHEDJfc* 

♦♦CALCULATIONS  WERE  STOPPED  AT 

(N=10000+T0TAL  NUMBER  OF  ISTOPS#)^* 

N=  10080.0010000  PHI=  .000010108  LLR=  -263.02 


SCHICK-UOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED^* 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+TOTAL  NUMBER  OF  ISTOPS#)*^ 

N=  10080.0010000  PHI=  .000000691  LLR=  -356.08 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  .0808674  K=  1.006034619  LLR=  -262.18 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.0130925  BETA=  8.081652568  LLR=  -265.35 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF5G 

EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL. 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 198. 

TOTAL  NUMBER  OF  INTERVALS  = 80. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORAMDA  SOFTWARE  RELIABILITY  MODEL 
N=  538.1000000  PHI  = .000579303  LLR=  -310.90 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

♦♦CONVERGENCE  WAS  NOT  REACHED** 

♦♦CALCULATIONS  WERE  STOPPED  AT 

<N=10000+T0TAL  NUMBER  OF  tSTOPS#)** 

N=  10198.1000000  PHI=  .000001698  LLR=  -599,78 

:| 

j MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

' N=  208.1000000  PHI=  .000009344  LLR=  -317.00 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .1015205  K=  1.000000225  LLR=  -263.00 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  .9783441  BETA=  7.429661570  LLR=  -643.07 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
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file:  EF5AH 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AG  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 432. 

TOTAL  NUMBER  OF  INTERVALS  = 131. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  1020.0010000  PHI=  .001696781  LLR=  -67.87 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  476.0010000  PHI=  .009180697  LLR=:  -43.09 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  1.7285712  K=  .998994010  LLR=  -67.87 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  2.1878506  BETA=  .677890384  LLR=  -52.64 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
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file:  EF5AH 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 432. 

TOTAL  NUMBER  OF  INTERVALS  = 131. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  1283.1000000  PHI  = .005056635  LLR=  -212.86 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  933.1000000  PHI=  .017684491  LLR=  -619.92 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  463.1000000  PHI=  .000807953  LLR==  -233.33 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  1.6178002  K=  1.000000225  LLR=  -67.97 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  2.0099102  BETA=  .718852616  LLR=  -205.50 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 

• ♦ 
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file:  EF5BH 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 292. 

TOTAL  NUMBER  OF  INTERVALS  = 130. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  AREi 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  146.0010000  PHI=  .026925030  LLR=  -45.56 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  141.0010000  PHI=  .025678361  LLR=  -258,64 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  13.1518735  K=  .972742772  LLR=  -26.77 

UEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  .8020556  BETA=  .457405985  LLR=  -54.28 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF5BH 

EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 292. 

TOTAL  NUMBER  OF  INTERVALS  = 130. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOU-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  409.2000000  PHI  = .015238470  LLR=  -110.42 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  392.2000000  PHI=  .013734957  LLR=  -620.39 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  307.2000000  PHI=  .000874744  LLR=  -261.64 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  1.6021067  K=  1.000000225  LLR= 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 
ALPHA=  .8699890  BETA=  .441868617  LLR= 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 


♦ ♦ 


-68.72 


-86.38 


file:  EF5CH 


EVALUATION  OF  GROUP  I SOFTWARE  RELIAPILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 80. 

TOTAL  NUMBER  OF  INTERVALS  = 49. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE.* 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  133.0010000  PHI=  .008114052  LLR=  -55.44 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  103.0010000  PHI=  .010094756  LLR=  -77.63 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  1.2835062  K=  .984494710  LLR=  -55.15 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.1348635  BETA=  1.181769416  LLR=  -55.76 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF5CH 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 80. 

TOTAL  NUMBER  OF  INTERVALS  = ‘49. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  127.1000000  PHI  = .017387951  LLR=  -62.77 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  111.1000000  PHI=  .019708765  LLR=  -164.79 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  84.1000,000  PHI=  .001817107  LLR=  -93.20 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .8692568  K=  .999999998  LLR=  -55.87 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.1905967  BETA=  1.101664197  LLR=  -91.51 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 


144 


^ • 


file:  EF5DH 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 


TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 170. 

TOTAL  NUMBER  OF  INTERVALS  = 75. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE! 


JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 


104.0010000  PHI=  .018365972  LLR= 


-63.79 


SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 


80.0010000  PHI=  .043824873  LLR= 


-71.40 


JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 


2.1701876  K= 


.983265420  LLR= 


-63.72 


WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.6347346  BETA=  .945031468  LLR= 


THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 
JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 


-77.41 


file:  efsdh 


EVALUATION  OF  GROUP  II  SOFTUARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 170. 

TOTAL  NUMBER  OF  INTERVALS  = 75. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTUARE  RELIABILITY  MODEL 
N*  205.1000000  PHI  = .025091666  LLR=  -98.18 

LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  175.1000000  PHI=  .054954685  LLR=  -274.08 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  171.1000000  PHI=  .001963483  LLR-  -114.08 

MODIFIED  GEOMETRIC  SOFTUARE  RELIABILITY  MODEL 
D=  1.0872455  K=  1.000000225  LLR=  -68.72 

WEIBULL  SOFTUARE  RELIABILITY  MODEL 

ALPHA*  1.7219281  BETA*  ,938994089  LLR=  -161.96 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTUARE  RELIABILITY  MODEL 
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file:  EF5EH 


EVALUATION  OF  GROUP  I SOFTWARE  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 223. 

TOTAL  NUMBER  OF  INTERVALS  = 92. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  136.0010000  PHI=  .014357752  LLR=  -72.22 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  118.0010000  PHI=  .024763425  LLR=  -99.55 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  2.5625911  K=  .984498442  LLR=  -70.82 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.2124251  BETA=  .892276525  LLR=  -75.56 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
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file:  EF5EH 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF'  ERRORS  TO  DATE  = 223. 

TOTAL  NUMBER  OF  INTERVALS  = 92. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  273.1000000  PHI  = .021394947  Ll.R=  -122.05 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  252.1000000  PHI=  .034917187  LLR=  -371.89 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  227.1000000  PHI=  .001293842  LLR=  -170.28 

MODIFIED  GEOMETRIC  MODEL 

D=  1.1776384  K=  1.000000225  LLR=  -76.95 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  1.2576628  BETA=  .867753088  LLR=  -174.28 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED.* 

MODIFIED  GEOMETRIC  MODEL 
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file:  EF5FH 


EVALUATION  OF  GROUP  II  SOFTWARE  RELIABILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 207. 

TOTAL  NUMBER  OF  INTERVALS  = 90. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 
N=  218,1000000  PHI  = .036308260  LLR=  -109.78 

LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  215.1000000  PHI=  .035881426  LLR=  -406.03 

MODIFIED-LIPOW-SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 
N=  207.1000000  PHI=  .001809664  LLR=  -198.62 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  1.1246817  K=  1,000000225  LLR=  -79.42 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  .8218078  BETA=  .657791428  LLR=  -144.34 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
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file:  EF5GH 


EVALUATION  OF  GROUP  I SOFTUARF.  RELIABILITY  MODELS 
REQUIRING  ONLY  TIME  TO  ERROR  AS  INPUT  DATA 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  = 198. 

TOTAL  NUMBER  OF  INTERVALS  = 76, 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

JELINSKI-MORANDA  SOFTWARE  RELIABILITY  MODEL 

N=  78.0010000  PHI=  .044198370  LLR=  -48.82 

SCHICK-WOLVERTON  SOFTWARE  RELIABILITY  MODEL 

N=  76.0010000  PHI=  .028922954  LLR=  -196.67 

JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 
D=  14.4139897  K=  .945673555  LLR=  -32.41 

WEIBULL  SOFTWARE  RELIABILITY  MODEL 

ALPHA=  .6378079  BETA=  .599887132  LLR=  -54.62 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

.JELINSKI-MORANDA  GEOMETRIC  SOFWARE  RELIABILITY  MODEL 


♦ • 


file:  EF5GH 


EVALUATION  OF  nF;OUP  II  SOFTUARE  RELIAHILITY  MODELS 
REQUIRING  TIME  TO  ERROR  AND  NUMBER  OF  ERRORS  PER  INTERVAL 

TOTAL  NUMBER  OF  ERRORS  TO  DATE  - 198. 

TOTAL  NUMBER  OF  INTERVALS  = 76. 

PARAMETER  ESTIMATIONS  FOR  THE  TESTED  MODELS  ARE: 

LIPOW-JELINSKI -MORANDA  SOFTUARE  RELIABILITY  MODEL 
N=  198.1000000  PHI  = .068513675  LLR=  -91.83 

LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  198.1000000  PHI=  .044056958  LLR=  -431.00 

MODIFIED-LIPOU-SCHICK-UOLVERTON  SOFTUARE  RELIABILITY  MODEL 
N=  198.1000000  PHI=  .002860647  LLR=  -185.42 

MODIFIED  GEOMETRIC  SOFTWARE  RELIABILITY  MODEL 
D=  .9447691  K=  1.000000225  LLR=  -80.32 

UEIBULL  SOFTUARE  RELIABILITY  MODEL 

ALPHA=  .6623245  BETA=  .391920693  LLR=  -62.85 

THE  LIKELIHOOD  RATIO  TEST  HAS  SELECTED: 

WEIBULL  SOFTUARE  RELIABILITY  MODEL 
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Append ix  D 

Program  Time-To-Fix  Input 


Contained  within  this  appendix  is  a portion  of  File5 
used  as  input  into  Program  Time-To-Fix,  which  was  used  to 
select  a distribution  for  the  software  time-to-fix  data 
files.  The  first  six  positions  are  the  time  it  took  to  fix 
the  software  error.  The  seventh  postion  is  the  criticality 
of  the  error  and  the  last  five  positions  indicate  the  error 
category  which  was  deemed  appropriate  for  this  particular 
software  problem  report. 


ram  Time-To-Fix 


i 


.84D6B060 

*84DBEi060 

.84CBB070 

.84DBB080 

.84CBB060 

.85CBB020 

.86DBB130 

.86CBB070 

♦87DBB070 

.87CBB070 

.88DBB140 

.88DBB120 

.88DBB060 

.88UBB070 

.88CBB060 

.88CBB140 

♦88CBB070 

♦90CBB070 

.90CBB060 

♦90CBB070 

.91DBB080 

♦92CBB080 

.92CBB070 

♦92CBB070 

♦92CBB070 

♦93DBB060 

.93DBB060 

.93CBB070 

.93DBB060 

»93CBB140 

♦94DBB0A0 

♦94DBB080 

.95CBB080 

♦95CBB140 

♦95CBB070 

.95DBB140 

♦95CBB060 

.95DBB070 

.95CBB070 

♦97CBB070 

.97DBB070 

.99DBB140 

♦99DBB140 
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Appendix  E 

Program  Time-To-Fix  * s Output 


The  next  three  pages  are  typical  outputs  from 
Program  Time-To-Fix.  The  first  is  the  result  of  using  file 
FF3  as  input  without  regards  to  error  categories.  The 
second  page  is  the  output  for  category  "KK”  of  file  FF5.  As 
can  be  seen  the  log-normal  distribution  was  selected. 
Finally,  the  third  listing  shows  what  happened  when  there 
was  only  one  data  point  entered  as  data. 
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INPUT  CONSIST'D  Sr  219  POINTS 
P»»T  J.  P*RA“ET€*  iSTIHATION 


OISTPIS'ITTON  PAPAKETiR  1 PARAMETER  2 


UNIFORM 

AU'^HA  9 

.5oecE»ia 

9ETA  « 

.4830E*02 

CWPONENTTAL 

TM^TA  * 

.530rE»G3 

MU  * 

.524-.E*ai 

wEiaULU 

“^fTA  » 

, 108e£*31 

alpha  » 

.E’44E*iJ1 

NORMAL 

,57i.4£*0t 

SXSMA  = 

.7139E.01 

LON-hoRMAL 

H<f  « 

.lt23E*Cl 

SIGhA  s 

.i:38£»0l 

r.AMMA 

Atcw4  s 

.9407E»C: 

9ETA  X 

.6i:SE*91 

LOGISTIC 

MU  » 

,5744e*Cl 

STJMA  s 

.n39E*31 

EXT  VALU'-SMALLEST 

.5SSFe*01 

Mija 

.ao57e*ei 

E*T  VIL'IE -LARGEST 

S?Ta  ■ 

.55SoE»Tl 

M'J  a 

.2532E*11 

PARETO 

K » 

.5C0CE*CJ 

A » 

. 5508E*  00 

LAPLACE 

THETA  * 

.4]03E«’C1 

PHI  a 

.ir44E*oi 

PAP*  11.  KaHOGOP.OV-SHIRNOV  TEST  CNAT  OVILUE  » .:72I5I 

OISTPI3UTION 

MAX  K3VALUE 

DECISION 

UKIFORH 

.6S922 

REJECT 

E*PO*e‘ITI  AL 

. .2153C 

REJECT 

Wf**'A.L 

.15207 

REJECT 

NO«NAl 

.22A71 

REJECT 

lcg-nornal 

.1J*62 

».£  JFCT 

CA*MA 

.13327 

REJECT 

LOGISTIC 

.20<>18 

REJECT 

C*T  VI LU* -smallest 

.25?3» 

REJECT 

EXT  VALUE -LAPSES! 

.23224 

REJECT 

PARETO  • 

.234T5 

PSJ'CT 

LAPLACE 

.22927 

REJECT 

•ART  111.  LIXLIM300  RATIO 

TEST. 

OISTP.IBUT  ION 

LN  CF  LINEL 

iMOOO  FO'tCTlOH 

UNIPO’M 

-945. 

ExeO»'ENTI  AL 

-562. 

WE TOOL L 

-«CS. 

NORMAL 

-741. 

LCG-NORNAL 

-594. 

GAMMA 

•sot. 

LOGISTIC 

-7  08. 

EXT  VAL'JP -smallest 

999939. 

EXT.  VALU'-LARSSST 

-S31. 

PAPETO 

-595. 

LAFLAC-e 

-RIO. 

PART  IV.  oecistou. 

NO  DISTRIBUTION  WAS  SELECTED 


I 


1 


*M»LYSTS  OY  StTE'-.QOy 
I*|P!JT  CONSISTED  0?  31  OOINTS 


MUT  I.  P4T4NETEY  ESTIHATTOM 
OISTRISJTIOM 


PAXANErEit  1 


PARANETER  2 


tIHTPORN 

ALPHA  a 

.13aCE*01 

BETA  a 

.8900E«02 

exponent  lAL 

theta  * 

.ijacE»ci 

-U  a 

.H29£*02 

NPIROLI. 

BETA.  » 

.942fE»00 

AL»mA  a 

.799TE»01 

NORN  At 

HU  * 

.1129E*C2 

sigma  a 

.1937E*02 

ta»;-^«RNAU 

HU  a 

.li*67£*01 

SIGPA  a 

.136l£*0l 

GANNA 

ALPHA  a 

.6SS2ETC9 

SETA  a 

.1727E*02 

LOCI  ST  i: 

HU  a 

.1129E*02 

SIGMA  a 

.19J7E»02 

EXT  VALUE-SHALLEST 

BETAa 

.1-32E*02 

HUa 

.19S6E*02 

EXT  VAUE-LARGFST 

BETA  a 

.li-32E*02 

HO  a 

.3923E»gi 

PARETO 

K a 

.i3aoE*ei 

A a 

.6B1SEY00 

LAPLACE 

theta  a 

.3000E*01 

PHI  a 

.0290E*01 

' II.  XOLNOGOPOV-SNIRNOX 

TEST  (HAX  OVALUE  a .19218) 

0ISTRI9JTI0N 

HAX  KStfALUE 

DECISION 

UNIPORN 

.65925 

REJECT 

exponential 

■ .35901 

REJECT 

WEI90LL 

.21435 

REJECT 

MO*NAl 

.29155 

REJECT 

LOC-'JORNAL 

.16674 

ACCEPT 

RAMNA 

.19290 

REJECT 

LOGISTIC 

.29323 

REJECT 

EX*  VAL'JE-SHALLEST 

.36165 

REJECT 

EXT  nAL'JE-LARGEST 

.23303 

REJECT 

PARETO 

.29032 

REJECT 

LAPLACE 

.36057 

REJECT 

PART  in.  LIKLIM003  PATIO  TEST. 


OISTRISUnON 


LR  OP  tIKELIHOOO  FUNCTION 


ONIPORH 

EXPO  WENT  lAL 

NEINV^L 

NOPNAL 

LCC-WORNAU 

CAHNA 

LOOT  STIC 

EXT  valje-smallest 
EXT  xAUE-LAPGEST 
PARETO 
UAPt^CE 


•tss. 

-103. 

-106. 

-134. 

-99. 

-104. 

-130. 

999999. 

-125. 

-00. 

-124. 


•APT  IW.  DECISION. 


THE  LIXUIHODO  peTtO  ANO  <0l.»05CIR0V-SH»,N0 7 TESTS  HAVE  SELECTEO  THE  LOC-NORHAL 
OISTRISUriO'l . PiRAXETEES  ARE 

HU  a .1467£*ei  SICHA  ■ .136tE*91 
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Append ix  F 

Sof tware  Error  Categories 


This  appendix  contains  a listing  of 
error  categories  used  by  the  contractors 
collection  procedures  (Ref  [ 301 7 ] : 45-50 ) . 


the  software 
during  error 
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ERROR  CATEGORIES 


CATEGORY 

ID 


AA07I 

AA072 


CaMPUTATICNAL  ERRORS 

TOTAL  NU.X5ER  OF  ENTRIES  CC.VrUTEO  INCORRECTLY 
PHYSICAL  OR  LOGICAL  ENTRY  NLT'BER  COMPUTED  INCORRECTLY 
WRONG  EQUATION  CR  CONVENTION  USED 
MATHEMATICAL  MODELING  PROBLEM 

RESULTS  OF  ARITHMETIC  CALCULATION  INACCURATE/NOT  AS  EXPECTED 

MIXED  MODE  ARITHMETIC  ERROR 

TIKE  CALCULATION  ERROR 

TIME  CONVERSION  ERROR 

TIME  TRUNCATION/ ROUNDING  ERROR 

SIGN  CONVENTION  ERROR 

UNITS  CONVERSION  ERROR 

VECTOR  CALCULATION  E'ROR 

CALCULATION  FAILS  TO  CONVERGE 

QUANTIZATION/TRUNCATION  ERROR 


LOGIC  ERRORS 

LIMIT  DETERMINATION  ERROR 

WRONG  LOGIC  BRANCH  TAKEN 

LOOP  EXITED  ON  WRONG  CYCLE 

INCOM,PLETE  PROCESS  INC- 

ENDLESS  LOOP  DURING  ROUTINE  OPERATION 

MISSING  LOGIC  CR  CONDITION  TEST 

INDEX  NOT  CHECKED 

FLAG  OR  SPECIFIC  CATA  VALUE  NOT  TESTED 
INCORRECT  LOGIC 
SEQUENCE  OF  ACTIVITIES  WRONG 
FILTERING  ERROR 

STATUS  CHECK/PROPAGATION  ERROR 
ITERATION  STEP  SIZE  INCORRECTLY  DETERMINED 
LOGICAL  CODE  PRODUCED  WRONG  RESULTS 
LOGIC  ON  WRONG  ROUTINE 

PHYSICAL  CHARACTERISTICS  OF  PROBLEM  TO  BE  SOLVED, 
OVERLOOKED,  OR  MISUNDERSTOOD 
LOGIC  NEEDLESSLY  COMPLEX 
INEFFICIENT  LOGIC 
EXCESSIVE  LOGIC 

STORAGE  REFERENCE  ERROR  (SOFTWARE  PROBLEM) 


I/O  ERRORS 
MISSING  OUTRITT 

OUTPin"  MISSING  DATA  ENTRIES  * 

ERROR  M-ESSAGE  NOT  OUTPUT 
ERROR  MESSAGE  GARBLED 

OUTPUT  OR  ERROR  .‘‘ESSAGE  NOT  COMPATIBLE  WITH  DESIGN 
DOCUMENTATION  (INCLUDING  GARBLED  OUTPUT) 

MISLEADING  OR  INACCURATE  ERROR  MESSAGE  TEXT 
OUTPUT  FCR-MAT  ERROR  (INCLUDING  WRONG  LOCATION) 

DUPLICATE  OR  EXCESSIVE  OUTPUT 
OUTPUT  FIELD  SIZE  INACEQUATE 

DEBUG  OUTPUT  PROBLEM  (RELATIVE  TO  DESIGN  DOCUMENTATION) 
LACK  OF  DEBUG  OUTPUT 
TOO  MUCH  DEBUG 
HEADER  OUTPUT  PROBLEM 
. OUTPUT  TAPE  FORMAT  ERROR 
OUTPUT  CARO  FORMAT  ERROR  ‘ 

ERROR  IN  PRINTER  CONTROL 
LINE  COUNT/PAGE  EJECT  ERROR 
NEEDED  OUTPUT  NOT  PROVIDED  IN  DESIGN 
INSUFFICIENT  OUTPUT  OPTIONS 


DATA  HANDLING  ERRORS 
VALID  INPUT  DATA  IMPROPERLY  SET/USED 
DATA  WRIHEN  IN  OR  READ  FROM  WRONG  DISK  LOCATION 
DATA  LOST/NOT  STORED 

DATA,  INDEX  OR  FLAG  NOT  SET  OR  SET/ INITIALIZED  INCORRECTLY 

NUMBER  OF  ENTRIES  SET  INCORRECTLY 

DATA,  INDEX  OR  FLAG  MODIFIED  OR  UPDATED  INCORRECTLY 

NUMBER  OF  ENTRIES  UPDATED  INCORRECTLY 

EXTRANEOUS  ENTRIES  GENERATED  (TABLE,  ARRAY,  ETC.) 

BIT  MANIPULATION  ERROR 

ERROR  USING  BIT  MC'DIFIER 

FLOATING  POINT/ INTEGER  CONVERSION  ERROR 

INTERNAL  VARIABLE  ERROR  (DEFINITION  OR  SET/USE) 

DATA  PACKING/UNPACKING  ERROR 

ROUTINE  LOOKING  FOR  DATA  IN  NON-EXISTENT  RECORD  . „ 
BOUNDS  VIOLATION 
DATA  CHAINING  ERROR 

DATA  OVERLOH  OR  OVERFLOW  PROCESSING  ERROR 

DPAn  FRpnp 

ALL  AVAILABLE  DATA  NOT  READ 
LONG  LITERAL  PROCESSING  ERROR 

SORT  ERROR  • ' - 

OVERLAY  ERROR 

SUBSCRIPTING  CONVENTION  ERROR 
DOUBLE  BUFFERING  ERROR 
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ERROR  CATEGORIES 


ERROR  CATEGORIES 


CATEGORY 

ID 

CATEGORIES 

JJOOO 

JJ010 

JJ020 

JJ030 

JJ040 

JJ050 

JJ060 

JJ070 

JJ080 

JJ090 

JJ100 

USER  INTERFACE  ERRORS 

OPERATIONS  RECUEST  OR  DATA  CARD/ROUTINE  INCOMPATABILITY 
MULTIPLE  PHYSICAL  CARD/LOGICAL  CARO  PROCESSING  ERROR 

INPUT  DATA  INTERPRETED  INCORRECTLY  BY  ROUTINE 

VALID  INPUT  DATA  REJECTED  OR  NOT  USED  BY  ROUTINE 

INPUT  DATA  REJECTED  BUT  USED 

INPUT  DATA  READ  BUT  NOT  USED 

ILLEGAL  INPUT  DATA  ACCEPTED  AND  PROCESSED 

LEGAL  INPUT  DATA  PROCESSED  INCORRECTLY 

POOR  DESIGN  IN  OPERATOR  INTERFACE 

INADEQUATE  INTERRUPT  AND  START  CAPABILITY 

KKOOO 

KK010 

KK011 

DATA  BASE  INTERFACE  ERRORS 

ROUTINE/DATA  BASE  INCOMPATIBILITY 

UNCOORDINATED  USE  OF  DATA  ELEMENTS  BY  MORE  THAN  ONE  USER 

LLOOO 

LL010 

LL020 

LL021 

LL022 

LL023 

LL024 

LL025 

LL030 

LL040 

LL050 

LL060 

LL070 

LL080 

USER  REQUESTED  CHANGES 

SIMPLIFIED  INTERFACE  AND/OR  CONVENIENCE 

NEW  AND/OR  ENHANCED  FUNCTIONS 

CPU 

DISK 

TAPE 

I/O 

CORE 

SECURITY 

NEW  HARDV.'ARE/OS  CAPABILITY 

INSTRUMENTATION  ■ - 

CAPACITY 

DATA  BASE  MANAGEMENT  AND  INTEGRITY 

EXTERNAL  PROGRAM  INTERFACE 

mooo 

MM010 

m020 

MM030 

HM040 

KM041 

HM050 

HM060 

PRESET  DATA  BASE  ERRORS 

DATA  OR  OPERATIONS  REQUEST  CARD  DESCRIPTIONS 

ERROR  MESSAGE  TEXT 

NOMINAL,  DEFAULT,  LEGAL,  MAX/MIN  VALUES 

PHYSICAL  CONSTANTS  AND  MODELING  PARAMETERS 

EPHEMERIS  parameters 

DICTIONARY  (BIT  STRING)  PARAMETERS 

MISSING  DATA  BASE  SETTINGS 

ERROR  CATEGORIES 


CATETORY 

10 

CATEGORIES 

NNOOO 

GLOBAL  VARIA3LE/C0MPC0L  DEFINITION  ERRORS 

NN010 

ITEMS  IN  WRONG  LOCATION  (WRONG  DATA  BLOCK) 

NN01I 

DEFINITION  SEQUENCE  ERROR 

- NN020 

DATA  DEFINITION  ERROR 

NN021 

TABLE  DEFINITION  INCORRECT  - - 

NN030 

LENGTH  OF  DEFINITION  INCORRECT  . - 

NN040 

COFT'tENTS  ERROR 

NN050 

DELETE  UNNEEDED  DEFINITIONS 

PPOOO 

RECURRENT  ERRORS 

PPOlO 

PROBLEM  REPORT  REOPENED 

PP020 

PROBLEM  REPORT  A DUPLICATE  OF  PREVIOUS  REPORT 

QQOOO 

DOCUMENTATION  ERRORS 

QQ010 

ROUTINE  LIMITATION 

QQ020 

OPERATING  PROCEDURES 

QQ030 

DIFFERENCE  BETWEEN  FLOW  CHART  AND  CODE 

QQ040 

TAPE  FORMAT 

QQ050 

DATA  CARD/CPEPATION  REQUEST  CARD  FORMAT 

QQ060 

ERROR  MESSAGE 

QQ070 

ROUTINE'S  FUNCTIONAL  DESCRIPTION 

QQ080 

OUTPUT  FORMAT 

QQ090 

DOCUMENTATION  NOT  CLEAR/NOT  COMPLETE  ' 

QQ100 

TEST  CASE  docu:;entation 

QQ110 

OPERATING  SYSTEM  DOCUMENTATION 

QQ120 

TYPO/EDITORIAL  ERROR/COSMETIC  CHANGE  ^ 

RROOO  REOUIREKENTS  COMPLIANCE  ERRORS 

RROlO  EXCESSIVE  RUN  TIME 

RR020  REQUIRED  CAPABILITY  OVERLOOKED  OR  NOT  DELIVERED  AT 

TIME  OF  REPORT 
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ERROR  CATEGORIES 


! 


1 


CATEGORY 

ID 

CATEGORIES 

ssooo 

UNIDENTIFIED  ERRORS 

nooo 

TTOIO 

TT020 

TT030 

n040 

no5o 

OPERATOR  ERROR 

TEST  EXECUTION  ERROR 

ROUTINE  COV.PILEO  AGAINST  WRONG  COMPOOL/KASTER  COfT^IN 

WRONG  DATA  BASE  USED 

WRONG  MASTER  CONFIGURATION  USED 

WRONG  TAPE(S)  USED 

uuooo 

UU010 

UU020 

UU030 

QUESTIONS  - . ■ 

DATA  BASE 

MASTER  CONFIGURATION 

ROUTINE  ' 1 

V 

HARDWARE 

1 

X 

1 

NON-REPRODUCIBLE 

References 


1,  Barlow,  R.  H.  and  F.  Porschan.  Ma thematical  Theory  of 
Reliability.  New  York:  John  Wiley  & and  Sons,  Inc, 
19^ 


2.  Caplen,  R.  A Practical  Approach  Reliability 

London,  England:  Business  Books  Limited,  1972. 


3.  Corcoran,  W.  J.,  H.  Weingarten, 

P.  W.  Zehna.  "Estimating  Reliability  After 

Corrective  Action,"  Management  Science,  10:  786-795 
(July  1964). 


4.  Dumonceaux,  R.  (St.  Johns  University),  C.  E.  Antle 
(The  Pennsylvania  State  University),  G.  Haas 
(Northrop  Corporation  and  West  Coast  University), 
"Likelihood  Ratio  Test  for  Discrimination  Between 
Two  Models  with  Unkown  Location  and  Scale 

Parameters,"  Air  Force  Aerospace  Research 

Laboratories,  Air  Force  Systems  Command,  United 
States  Air  Force,  Contract  No.  F336 1 5-7  1 -C- 1 1 69 . 


5.  Dryer,  A.  R.  "Hypothesis  Testing  Procedures 

for  Seperate  Families  of  Hypotheses,"  Journal  of  the 
American  Statistical  Association,  69:  1 40- 1 45  TMarch 
19'74). 


6.  Farnan,  D.  N.  (Defense  Systems  Management 

School).  "Relaible  Computer  Software.  What  Is  It  and 
How  to  Get  It,"  ^ House  Report , November  1975.  (AD- 
A026988) . 


7.  Fries,  M.  J.  "Software  Error  Data  Acquisition," 

House  Report,  Boeing  Aerospace  Company,  April 
1977.  (RAD‘C-TR-77-130) . 


8.  Jelinski,  Z.,  P.  Moranda  "Software 

Reliability  Research,"  published  in  Statistical 
Computer  Performance  Ev aluation , edited  by 

W.  Freiberger.  New  York,  New  York:  Academic  Press, 
1972. 


166 


I ' 9.  La  Padula,  L.  J.  "Engineering  of  Quality  Software 
f Systems  (Software  Relaibility  Modeling  and 
* Measurement  Techniques),"  In  House  Report , Mitre 
: Corporation,  January  1975.  TAD-A007773TT 


10.  Lilliefors,  H.  W.  "On  the  Kolmogorov-Smirnov  Test  for 
Normality  With  Mean  and  Variance  Unkown," 
American  Statistical  Association  Journal,:  399-402 
(June  196'7). 


11.  Lipow,  M.  "Maximum  Likelihood  Estimation  of 

Parameters  of  a Software  Time-To-Failure 
Distribution,"  TRW  Systems  Group,  TRW  Report  No. 
2260. 1 .9-73D-15  (RW1),  June  1973. 


12.  Lipow,  M.  "Some  Variations  of  a Model  for 
Software  Time-To-Failure,  TRW  Systems  Group, 
Correspondence  ML-74-2260 . 1 . 9-2 1 , August  1974. 


13.  Massey,  F.  J.  "The  Kolmogorov-Smirnov  Test 

For  Goodness  of  Fit,"  American  Statistical 
Association  Journal , : 68-78  (March  1951). 


14.  Moranda,  P.  3.  "Prediction  of  Software 

Reliability  During  Debugging,"  Proceedings  1975 
Annua_l  Reliabil  ity  and  Maintainability  Symposium , 
"BaYhington,  D.  C.  , January  19T5. 


15.  Moranda,  P.  B.  "Probability-Based  Models  for  the 
Failures  During  Burn-In  Phase,"  Joint  National 
Meeting  ORSA/TIMS,  Las  Vegas,  New  Mexico,  November 
1975.'  CAD-A0207'M)  . 


16.  Nelson  E.  C.  "A  Statistical  Basis  for 

Software  Reliability  Assessment,"  TRW  Software 

S.eri.^,  T^-^-;^-pJ,  March  1973. 


17.  Randell,  B.  "System  Structure  for  Software 

Fault  Tolerance,"  IEEE  Software  Engineering,  SE1,2: 
220-232  (June  1975). 


Regul  inski,  T.  L.  Program  Moj^l  - A computer  program 
written  for  hypothesis  testing  of 

probability  density  functions.  Given  at  the  Air 
Force  Institute  of  Technology  Wright  Patterson  AFB, 
Jh.o,  1977. 


167 


(»■ . 


29. 


William , H . E . , 

A.  A.  Beaureguard , P.  Hilcoff, 
Reliability:  A Raytheon  Project 


Report,  Raytheon  Company,  June 
188). 


T.  A.  James, 
"Software  Systems 
History,"  In  House 
1977.  (RADC-TR-77- 


30.  Wolverton,  R.  W.  and  G.  J.  Schick,  "Assessment 
of  Software  Reliability,"  TRW  Software  Series ,TRW- 
SS-73-0^ , September  1972. 


Vita 


Steve  G.  Castle  was  born  on  2 December  19^8  in 
Ilion,  New  York.  After  graduating  from  Ilion  Central  High 
School  in  1966,  he  continued  his  education  at  Utica  College 
of  Syracuse  University,  Utica,  New  York,  from  which  he  was 
graduated  with  a Bachelor  of  Arts  degree  in  Mathematics  on 
6 June  1970.  Upon  graduation,  he  enlisted  in  the  Air  Force 
and  served  as  an  Accounting  and  Finance  Specialist  at  Myrtle 
Reach  AFB,  South  Carolina.  In  1972  he  attended  Officer 
Training  School  and  was  commissioned  to  be  a Second 
Lieutenant  on  13  October  1972.  From  1972  to  1976  he  served 
as  a Computer  Programmer  and  System  Analyst  at  Headquarters 
j'.jr  Training  Command,  Randolph  AFB,  Texas  for  which  he 
received  the  Air  Force  Commendation  Medal.  He  was  assigned 
zo  the  Air  Force  Institute  of  Technology  in  October,  1976. 

Permanent  address:  147  Oliver  Drive 

New  Smyrna  Beach,  Fla. 

32069 


UNCLASSIFIED 

SECUBfTY  classification  OF  THIS  PAGE  'Wfion  Dtim  Enterfd) 


^ REPORT  DOCUMENTATION  PAGE 


EA>oit^'m>MeER 


3 A^p  READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

2.  GOVT  ACCESSION  NO.  3.  RECIPIENT'S  CATALOG  NUMBER 


I^AFIT/GCS/EE/78-2  | 

4.  T4TLfc(lMi<  lluUII^Sr-"*^"^— • f 

" '•  ^FTOARE  ^LIABILITY;  JIODELLING  I 

f - I tihe-to-eMor  and  TIHE-TO  -FIX^  1 


J Steve  G./ Castle 


9.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 


Air  Force  Institute  of  Technology  (AFIT-EN) 
Wright  Patterson  AFB,  Ohio  45433 


11.  CONTROLLING  OFFICE  NAME  AND  ADDRESS 

Rome  Air  Development  Center  (RADC/ISIS)  ( 

Griffiss  AFB,  New  York 


14.  MONITORING  AGENCY  NAME  & ADDRESSri/ (ii7/erenf  from  Controltint  Office) 


S.  TYPE  OF  REPORT  & PERIOD  COVERED 


e.  CONTRACT  OR  GRANT  NUMBERf«J 


10.  PROGRAM  ELEMENT.  PROJECT.  TASK 
AREA  a WORK  UNIT  NUMBERS 


REPORT  DATE 


Unclassified 


ISa.  DECLASSIFICATION/ DOWNGRADING 
SCHEDULE 


16.  DISTRIBUTION  STATEMENT  (of  thia  Report) 


Approved  for  public  release;  distribution  unlimited 


'7.  distribution  statement  (of  the  abstract  entered  fn  Btoek  30,  If  different  from  Report) 


18.  SUPPLEMENTARY  NOTES 


^Js^^ved  ^^^»^ub^^*<^re^e^es ; lAW  AFR 

JERRAL  F.  GU^S,  Captain,  USAF 
Directiir  of  Information 


190-17 


19.  KEY  WORDS  (Conthwe  on  rev«rae  a/de  if  necoosary  and  identify  by  block  number) 

Software  Reliability 
Reliability 
Computer  Software 
Modelling 


20.  abstract  (Continue  on  reveree  aide  if  neceeeary  end  identify  by  block  number)  , . . • ! 

Various  competing  software  reliability  models  were  tested  utilizing  time-to- 
error  data  extracted  from  a large  data  base  provided  by  Rome-Air  Development 
Center  (RADC),  Griffiss  Air  Force  Base,  New  York.  Only  those  models  for  which 
appropriate  information  was  available  from  the  RADC  data  base  were  used.  Each 
model  that  was  tested  had  its  paramters  estimated  and  discrimination  between  the 
models  was  performed  by  the  Likelihood  Ratio  Test.  Code  for  the  computer  program 
used  in  parameter  estimation  and  Likelihood  Ratio  Testing  is  included.  The  same 
large  data  base  was  also  used  in  determination  of  the  probability  functions  ; 


FORM 
I JAN  73 


EDITION  OF  1 NOV  65  IS  OBSOLE: 


UNCLASSIFIED 

y^LCURITY  CLASSIFICATION  OF  THIS  PAGE  fHhen  Oaf*  Enlcr.cf) 

^ Lq,' 


UNCLASSIFIED 


security  classification  of  this  PAGEfHTian  Palm  Enlarad) 


'SWerning  the  random  variable  time-to-fix.  Time-to-fix  data  was  extractea  rrom 

data  base  and  was  sorted  by  error  categories.  Eleven  of  the  more  common 
probability  density  functions,  along  with  the  Kolmogorov- Smirnov  Test  and  the 
Likelihood  Ratio  Test,  were  used  to  isolate  the  distribution  that  best  modelled 
software  time-to-fix  for  each  particular  data  set. 


security  classification  of  this  PAOEfHTian  Oar*  Enletad) 


